Planet Grep

Planet'ing Belgian FLOSS people

Planet Grep is maintained by Wouter Verhelst. All times are in UTC.

September 18, 2017

The post A proposal for cryptocurrency addresses in DNS appeared first on ma.ttias.be.

By now it's pretty clear that the idea of a cryptocurrency probably isn't going away. It might not be Bitcoin or Litecoin, it might not have the same value as it does today, but the concept of cryptocurrency is here to stay: digital money.

Just like the beginning of IP addresses, using them raw was fine at first. But with crypto, you get long hexadecimal strings that truly no one can remember by heart. It's far from user friendly.

It's like trying to remember that 2a03:a800:a1:1952::ff is the address for this site. Doesn't work very well, does it? It's far easier to say ma.ttias.be than the cryptic representation of IPv6.

I think we need something similar for cryptocurrencies. Something independent and -- relatively -- secure. So here's my proposal I came up with in the car on the way home.

Example: cryptocurrency in DNS

Here's the simplest example I can give.

$ dig ma.ttias.be TXT | sort
ma.ttias.be.	3600	IN    TXT   "10 btc:1AeCyEczAFPVKkvausLSQWP1jcqkccga9m"
ma.ttias.be.	3600	IN    TXT   "10 ltc:Lh1TUmh2WP4LkCeDTm3kMX1E7NQYSKyMhW"
ma.ttias.be.	3600	IN    TXT   "20 eth:0xE526E2Aecb8B7C77293D0aDf3156262e361BfF0e"
ma.ttias.be.	3600	IN    TXT   "30 xrp:rDsbeomae4FXwgQTJp9Rs64Qg9vDiTCdBv"

Cryptocurrency addresses get published as TXT records to a domain of your choosing. Want to receive a payment? Simple say "send it to ma.ttias.be", the client will resolve that TXT record and the accompanying addresses and use the priority field as a guideline for choosing which address to pick first.

Think MX records, but implemented as TXT. The lower the priority, the more preferred it is.

The TXT format explained

A TXT format can contain pretty much anything, so it needs some standardization in order for this to work. Here's my proposal.

[priority] space [currency]:[address]

Let's pick the first result as an example and tear it down.

$ dig ma.ttias.be TXT | sort | head -n 1
ma.ttias.be.	3600	IN    TXT   "10 btc:1AeCyEczAFPVKkvausLSQWP1jcqkccga9m"

Translates to;

  • 10: the priority. The lower, the bigger its preference.
  • btc: preferred currency is btc, or Bitcoin.
  • 1AeCyEczAFPVKkvausLSQWP1jcqkccga9m: the btc address to accept payments.

Simple, versatile format.

The priority allows for round robin implementations, if you wish to diversify your portfolio. Adding multiple cryptocurrency allows the sender the freedom to choose which currency he/she prefers, while still honoring your priority.

Technically, I published 2 records with a priority of 10. It's up to the sender to determine which currency he/she prefers, if it's available to them. If it isn't, they can move down the chain & try other addresses published.

It means only addresses on which you want to receive currency should ever be posted as DNS records.

DNSSEC required

To avoid DNS cache poisoning or other man-in-the-middle attacks, DNSSEC would have to be a hard requirement in order to guarantee integrity.

This should not be optional.

If smart people every end up implementing something like this, an additional GPG/PKI like solution might be added for increased security, by signing the addresses once more.

Why not a dedicated resource record?

For the same reason the SPF resource record went away and was replaced by a TXT alternative: availability.

Every DNS server and client already understands TXT records. If we have to wait for both servers, clients and providers to implement something like a ICO resource record, it'll take ages. Just look at the current state of CAA records, only a handful of providers offer it, even though it's a mandatory CA thing already.

There are already simpler naming schemes for cryptocurrency!

Technically, yes, but they all have a deep flaw: you have to trust someone else's system.

There's BitAlias, onename, ens for ethereum, okTurtles, ... and they all build on top of their own, custom system.

But it turns out, we already have a name-translation-system called DNS, we'd be far better of implementing a readable cryptocurrency variant in DNS than in someone else's closed system.

The validator regex

With the example given above, it can easily be validated with the following regex.

([0-9]+) ([a-z]{3}):([a-zA-Z0-9]+)

And it translates to;

  • group #1: the priority
  • group #2: the currency
  • group #3: the address

Now, who's going to make this an RFC? I certainly won't, I've got too many things to do already.

The post A proposal for cryptocurrency addresses in DNS appeared first on ma.ttias.be.

I published the following diary on isc.sans.org: “Getting some intelligence from malspam“.

Many of us are receiving a lot of malspam every day. By “malspam”, I mean spam messages that contain a malicious document. This is one of the classic infection vectors today and aggressive campaigns are started every week. Usually, most of them are blocked by modern antivirus or anti-spam but these files could help us to get some intelligence about the topic used by attackers to fool their victims. By checking the names of malicious files (often .rar, .gip or .7r archives), we found classic words like ‘invoice’, ‘reminder’, ‘urgent’, etc… [Read more]

[The post [SANS ISC] Getting some intelligence from malspam has been first published on /dev/random]

September 17, 2017

The post Chrome to force .dev domains to HTTPS via preloaded HSTS appeared first on ma.ttias.be.

tl;dr: one of the next versions of Chrome is going to force all domains ending on .dev (and .foo) to be redirected to HTTPs via a preloaded HTTP Strict Transport Security (HSTS) header.


This very interesting commit just landed in Chromium:

Preload HSTS for the .dev gTLD.

This adds the following line to Chromium's preload lists;

{ "name": "dev", "include_subdomains": true, "mode": "force-https" },
{ "name": "foo", "include_subdomains": true, "mode": "force-https" },

It forces any domain on the .dev gTLD to be HTTPs.

Wait, there's a legit .dev gTLD?

Yes, unfortunately.

It's been bought by Google as one of their 100+ new gTLDs. What do they use it for? No clue. But it's going to cause a fair bit of confusion and pain to webdevelopers.

The .dev gTLD has nameservers and is basically like any other TLD out there, we as developers just happen to have chosen that name as a good placeholder for local development, too, overwriting the public DNS.

$ dig +trace dev. NS
dev.			172800	IN	NS	ns-tld4.charlestonroadregistry.com.
dev.			172800	IN	NS	ns-tld5.charlestonroadregistry.com.
dev.			172800	IN	NS	ns-tld3.charlestonroadregistry.com.
dev.			172800	IN	NS	ns-tld2.charlestonroadregistry.com.
dev.			172800	IN	NS	ns-tld1.charlestonroadregistry.com.

Google publishes some of their domains on there, too;

$ dig +trace google.dev A
google.dev.		3600	IN	A	127.0.53.53

So yes, it's a legit TLD.

Consequences of redirecting .dev to HTTPS

A lot of (web) developers use a local .dev TLD for their own development. Either by adding records to their /etc/hosts file or by using a system like Laravel Valet, which runs a dnsmasq service on your system to translate *.dev to 127.0.0.1.

In those cases, if you browse to http://site.dev, you'll be redirect to https://site.dev, the HTTPS variant.

That means your local development machine needs to;

  • Be able to serve HTTPs
  • Have self-signed certificates in place to handle that
  • Have that self-signed certificate added to your local trust store (you can't dismiss self-signed certificates with HSTS, they need to be 'trusted' by your computer)

Such fun.

What should we do?

With .dev being an official gTLD, we're most likely better of changing our preferred local development suffix from .dev to something else.

There's an excellent proposal to add the .localhost domain as a new standard, which would be more appropriate here. It would mean we no longer have site.dev, but site.localhost. And everything at *.localhost would automatically translate to 127.0.0.1, without /etc/hosts or dnsmasq workarounds.

Alternatively, if you're looking for a quick "search and replace" alternative for existing setups, consider the .test gTLD, which is a reserved name by IETF for testing (or development) purposes.

I do hope the Chromium team reconsiders the preloaded HSTS as it's going to have rather big implications for local webdevelopment.

The post Chrome to force .dev domains to HTTPS via preloaded HSTS appeared first on ma.ttias.be.

September 15, 2017

Last week, Equifax, one of the largest American credit agencies, was hit by a cyberattack that may have compromised the personal data of nearly 143 million people, including name, address, social security numbers, birth dates and more. The forfeited information reveals everything required to steal someone's identity or to take out a loan in someone else's name. Considering that the current US population is 321 million, this cyberattack is now considered to be one of the largest and most intrusive breaches in US history.

It's Equifax that is to blame, not open-source

As Equifax began to examine how the breach occurred, many unsubstantiated reports and theories surfaced in an attempt to pinpoint the vulnerability. One such theory targeted Apache Struts as the software responsible for the breach. Because Apache Struts is an open-source framework used for developing Java applications, this resulted in some unwarranted open-source shaming.

Yesterday, Equifax confirmed that the security breach was due to an Apache Struts vulnerability. However, here is what is important; it wasn't because Apache Struts is open-source or because open-source is less secure. Equifax was hacked because the firm failed to patch a well-known Apache Struts flaw that was disclosed months earlier in March. Running an old, insecure version of software — open-source or proprietary — can and will jeopardize the security of any site. It's Equifax that is to blame, not open-source.

The importance of keeping software up-to-date

The Equifax breach is a good reminder of why organizations need to remain vigilant about properly maintaining and updating their software, especially when security vulnerabilities have been disclosed. In an ideal world, software would update itself the moment a security patch is released. WordPress, for example, offers automatic updates in an effort to promote better security, and to streamline the update experience overall. It would be interesting to consider automatic security updates for Drupal (just for patch releases, not for minor or major releases).

In absence of automatic updates, I would encourage users to work with PaaS companies that keep not only your infrastructure secure, but also your Drupal application code. Too many organizations underestimate the effort and expertise it takes to do it themselves.

At Acquia, we provide customers with automatic security patching of both the infrastructure and Drupal code. We monitor our customers' sites for intrusion attempts, DDoS attacks, and other suspicious activity. If you prefer to do the security patching yourself, we offer continuous integration or continuous delivery tools that enable you to get security patches into production in minutes rather than weeks or months. We take pride in assisting our customers to keep their sites current with the latest patches and upgrades; it's good for our customers and helps dispel the myth that open-source software is more susceptible to security breaches.

Last week, Equifax, one of the largest American credit agencies, was hit by a cyberattack that may have compromised the personal data of nearly 143 million people, including name, address, social security numbers, birth dates and more. The forfeited information reveals everything required to steal someone's identity or to take out a loan in someone else's name. Considering that the current US population is 321 million, this cyberattack is now considered to be one of the largest and most intrusive breaches in US history.

It's Equifax that is to blame, not open-source

As Equifax began to examine how the breach occurred, many unsubstantiated reports and theories surfaced in an attempt to pinpoint the vulnerability. One such theory targeted Apache Struts as the software responsible for the breach. Because Apache Struts is an open-source framework used for developing Java applications, this resulted in some unwarranted open-source shaming.

Yesterday, Equifax confirmed that the security breach was due to an Apache Struts vulnerability. However, here is what is important; it wasn't because Apache Struts is open-source or because open-source is less secure. Equifax was hacked because the firm failed to patch a well-known Apache Struts flaw that was disclosed months earlier in March. Running an old, insecure version of software — open-source or proprietary — can and will jeopardize the security of any site. It's Equifax that is to blame, not open-source.

The importance of keeping software up-to-date

The Equifax breach is a good reminder of why organizations need to remain vigilant about properly maintaining and updating their software, especially when security vulnerabilities have been disclosed. In an ideal world, software would update itself the moment a security patch is released. WordPress, for example, offers automatic updates in an effort to promote better security, and to streamline the update experience overall. It would be interesting to consider automatic security updates for Drupal (just for patch releases, not for minor or major releases).

In absence of automatic updates, I would encourage users to work with PaaS companies that keep not only your infrastructure secure, but also your Drupal application code. Too many organizations underestimate the effort and expertise it takes to do it themselves.

At Acquia, we provide customers with automatic security patching of both the infrastructure and Drupal code. We monitor our customers' sites for intrusion attempts, DDoS attacks, and other suspicious activity. If you prefer to do the security patching yourself, we offer continuous integration or continuous delivery tools that enable you to get security patches into production in minutes rather than weeks or months. We take pride in assisting our customers to keep their sites current with the latest patches and upgrades; it's good for our customers and helps dispel the myth that open-source software is more susceptible to security breaches.

September 14, 2017

The post Laravel Horizon: requires ext-posix, missing from CentOS appeared first on ma.ttias.be.

Here's what I ran into when I tried to install a project that required laravel/horizon via Composer.

$ composer install
Loading composer repositories with package information
Installing dependencies (including require-dev) from lock file
Your requirements could not be resolved to an installable set of packages.

  Problem 1
    - Installation request for laravel/horizon v1.0.3 -> satisfiable by laravel/horizon[v1.0.3].
    - laravel/horizon v1.0.3 requires ext-posix * -> the requested PHP extension posix is missing from your system.
...

The error message requires ext-posix * -> the requested PHP extension posix is missing from your system is actually confusing. On CentOS, there's no PHP package called 'posix', even though the PHP module is called POSIX.

$ php -m | grep posix

(If that doesn't return any results, the posix extension is missing.)

On CentOS, the package you're looking for is called process, as it contains a set of functions/methods to help with creating child processes, sending signals, parsing ID/GIDs, ...

If you're using the IUS repositories on CentOS/Red Hat, you can install them via;

$ yum install php71u-process

Afterwards, if you run composer again, it'll work. To verify if the posix extension is installed properly, run php -m again.

 $ php -m | grep posix
posix

Now, the posix extension is installed.

The post Laravel Horizon: requires ext-posix, missing from CentOS appeared first on ma.ttias.be.

I published the following diary on isc.sans.org: “Another webshell, another backdoor!“.

I’m still busy to follow how webshells are evolving… I recently found another backdoor in another webshell called “cor0.id”. The best place to find webshells remind pastebin.com[1]. When I’m testing a webshell, I copy it in a VM located on a “wild Internet” VLAN in my home lab with, amongst other controls, full packet capture enabled. This way, I can spot immediately is the VM is trying to “phone home” to some external hosts. This was the case this time! [Read more]

 

[The post [SANS ISC] Another webshell, another backdoor! has been first published on /dev/random]

The post Presentation: Code Obfuscation, PHP shells & more appeared first on ma.ttias.be.

"What hackers do once they get past your code."

I gave this talk a while back and refreshed it a bit for DrupalCamp Antwerpen last Friday. It's a very fun topic where I get to show the results of a compromised website or server, from a hosting point of view.

The slides are pretty self-explanatory, but last time I checked there was also a video recording of the talk. If that makes it online, I'll make sure to add it here.

The post Presentation: Code Obfuscation, PHP shells & more appeared first on ma.ttias.be.

September 13, 2017

Last year, Matthew Tift and I examined Drupal.org's commit data to understand who develops Drupal, how much of that work is sponsored, and where that sponsorship comes from. We published our analysis in a blog post called "Who Sponsors Drupal Development?". A year later, I wanted to present an update. This year's report will also cover additional data, including gender and geographical diversity, and project sponsorship.

Understanding how an open-source project works is important because it establishes a benchmark for project health and scalability. Scaling an open-source project is a difficult task. As an open-source project's rate of adoption grows, the number of people that benefit from the project also increases. Often the open-source project also becomes more complex as it expands, which means that the economic reward of helping to improve the project decreases.

A recent article on the Bitcoin and Ethereum contributor communities illustrates this disparity perfectly. Ethereum and Bitcoin have market capitalizations valued at $30 billion and $70 billion, respectively. However, both projects have fewer than 40 meaningful contributors, and contribution isn't growing despite the rising popularity of cryptocurrency.

Number of Bitcoin contributors between 2010 and 2017According to Bitcoin's GitHub data, Bitcoin has less than 40 active contributors.
Number of Ethereum contributors between 2014 and 2017According to Ethereum's GitHub data, Ethereum has less than 20 active contributors.

Drupal, by comparison, has a diverse community of contributors. In the 12-month period between July 1, 2016 to June 30, 2017 we saw code contributions on Drupal.org from 7,240 different individuals and 889 different companies. This does not mean that Drupal is exempt from the challenges of scaling an open-source project. We hope that this report provides transparency about Drupal project development and encourages more individuals and organizations incentive to contribute. We also will highlight areas where our community can and should do better.

What is the Drupal.org credit system?

In the spring of 2015, after proposing ideas for giving credit and discussing various approaches at length, Drupal.org added the ability for people to attribute their work to an organization or customer in the Drupal.org issue queues. Maintainers of Drupal modules, themes and distributions can award issues credits to people who help resolve issues with code, translations, documentation, design and more.

Example issue credit on drupal orgA screenshot of an issue comment on Drupal.org. You can see that jamadar worked on this patch as a volunteer, but also as part of his day job working for TATA Consultancy Services on behalf of their customer, Pfizer.

Credits are a powerful motivator for both individuals and organizations. Accumulating credits provides individuals with a way to showcase their expertise. Organizations can utilize credits to help recruit developers or to increase their visibility in the Drupal.org marketplace.

While the benefits are evident, it is important to note a few of the limitations in Drupal.org's current credit system:

  • Contributing to issues on Drupal.org is not the only way to contribute. Other activities, such as sponsoring events, promoting Drupal, and providing help and mentorship, are important to the long-term health of the Drupal project. Many of these activities are not currently captured by the credit system. For this post, we chose to only look at code contributions.
  • We acknowledge that parts of Drupal are developed on GitHub and therefore aren't fully credited on Drupal.org. The actual number of contributions and contributors could be significantly higher than what we report.
  • Even when development is done on Drupal.org, the credit system is not used consistently; because using the credit system is optional, a lot of code committed on Drupal.org has no or incomplete contribution credits.
  • Not all code credits are the same. We currently don't have a way to account for the complexity and quality of contributions; one person might have worked several weeks for just one credit, while another person might receive a credit for ten minutes of work. In the future, we should consider issuing credit data in conjunction with issue priority, patch size, etc. We can also reduce the need for trivial credits by automating patch rerolls and automating coding style fixes.

Who is working on Drupal?

For our analysis we looked at all the issues that were marked "closed" or "fixed" in the 12-month period from July 1, 2016 to June 30, 2017. What we learned is that there were 23,238 issues marked "closed" or "fixed", a 22% increase from the 19,095 issues in the 2015-2016 period. Those 23,238 issues had 42,449 issue credits, a 30% increase from the 32,711 issue credits recorded in the previous year. Issue credits against Drupal core remained roughly the same year over year, meaning almost all of this growth came from increased activity in contributed projects. This is no surprise. Drupal development is cyclical, and during this period of the Drupal 8 development cycle, most of the Drupal community has been focused on porting modules from Drupal 7 to Drupal 8. Of the 42,449 issue credits reported this year, 20% (8,619 credits) were for Drupal core, while 80% (33,830 credits) went to contributed themes, modules and distributions.

Compared to the previous year, we also saw an increase in both the number of people contributing and the number of organizations contributing. Drupal.org received code contributions from 7,240 different individuals and 889 different organizations.

Contributions by individuals vs organizationsThe number of individual contributors is up 28% year over year and the number of organizations contributing is up 26% year over year.

While the number of individual contributors rose, a relatively small number of individuals still do the majority of the work. Approximately 47% of individual contributors received just one credit. Meanwhile, the top 30 contributors (the top 0.4%) account for over 17% of the total credits, indicating that these individuals put an incredible amount of time and effort in developing Drupal and its contributed projects:

RankUsernameIssues
1jrockowitz537
2dawehner421
3RenatoG408
4bojanz351
5Berdir335
6mglaman334
7Wim Leers332
8alexpott329
9DamienMcKenna245
10jhodgdon242
11drunken monkey238
12naveenvalecha196
13Munavijayalakshmi192
14borisson_191
15yongt9412189
16klausi185
17Sam152184
18miro_dietiker182
19Pavan B S180
20ajay_reddy176
21phenaproxima172
22sanchiz162
23slashrsm161
24jhedstrom155
25xjm151
26catch147
27larowlan145
28rakesh.gectcr141
29benjy139
30dhruveshdtripathi138

Out of the top 30 contributors featured, 19 were also recognized as top contributors in our 2015-2016 report. These Drupalists' dedication and continued contribution to the project has been crucial to Drupal's development. It's also exciting to see 11 new names on the list. This mobility is a testament to the community's evolution and growth.

Next, we looked at both the gender and geographic diversity of Drupal.org code contributors. While these are only two examples of diversity, this is the only available data that contributors can choose to share on their Drupal.org profiles. The reported data shows that only 6% of the recorded contributions were made by contributors that identify as female, which indicates a steep gender gap. Like in most open-source projects, the gender imbalance in Drupal is profound and underscores the need to continue fostering diversity and inclusion in our community.

Contributions by genderThe gender representation behind the issue credits. Only 6% of the recorded contributions are by women.
When measuring geographic diversity, we saw individual contributors from 6 different continents and 116 different countries:
Contributions by continent
Contributions by countryThe top 20 countries from which contributions originate. The data is compiled by aggregating the countries of all individual contributors behind each commit. Note that the geographical location of contributors doesn't always correspond with the origin of their sponsorship. Wim Leers, for example, works from Belgium, but his funding comes from Acquia, which has the majority of its customers in North America.

How much of the work is sponsored?

Drupal is used by more than one million websites. The vast majority of the individuals and organizations behind these Drupal websites never participate in the development of the project. They might use the software as it is or might not feel the need to help drive its development. We have to provide more incentive for these individuals and organizations to contribute back to the project.

Issue credits can be marked as "volunteer" and "sponsored" simultaneously (shown in jamadar's screenshot near the top of this post). This could be the case when a contributor does the minimum required work to satisfy the customer's need, in addition to using their spare time to add extra functionality.

While Drupal started out as a 100% volunteer-driven project, today the majority of the code on Drupal.org is sponsored by organizations. Only 11% of the commit credits that we examined in 2016-2017 were "purely volunteer" credits (4,498 credits), in stark contrast to the 46% that were "purely sponsored". In other words, there were four times as many "purely sponsored" credits as "purely volunteer" credits.

A few comparisons with the 2015-2016 data:

  • The credit system is used more. Between July 1, 2015 and June 30, 2016, 37% of all credits had no attribution while in the more recent period between July 1, 2016 to June 30, 2017, only 28% of credits lacked attribution. More people have become aware of the credit system, the attribution options, and their benefits.
  • Sponsored credits are growing faster than volunteer credits. Both "purely volunteer" and "purely sponsored" credits grew, but "purely sponsored" credits grew faster. There are two reasons why this could be the case: (1) more contributions are sponsored and (2) organizations are more likely to use the credit system compared to volunteers.
Contributions by volunteer vs sponsored

No data is perfect, but it feels safe to conclude that most of the work on Drupal is sponsored. At the same time, the data shows that volunteer contribution remains very important to Drupal. Maybe most importantly, while the number of volunteers and sponsors has grown year over year in absolute terms, sponsored contributions appear to be growing faster than volunteer contributions. This is consistent with how open source projects grow and scale.

Who is sponsoring the work?

Now that we have established that most of the work on Drupal is sponsored, we want to study which organizations contribute to Drupal. While 889 different organizations contributed to Drupal, approximately 50% of them received four credits or fewer. The top 30 organizations (roughly the top 3%) account for about 48% of the total credits, which implies that the top 30 companies play a crucial role in the health of the Drupal project. The graph below shows the top 30 organizations and the number of credits they received between July 1, 2016 and June 30, 2017:

Top 30 organizations contributing to DrupalThe top 30 contributing organizations based on the number of Drupal.org commit credits.

While not immediately obvious from the graph above, different types of companies are active in Drupal's ecosystem:

Category Description
Traditional Drupal businesses Small-to-medium-sized professional services companies that make money primarily using Drupal. They typically employ fewer than 100 employees, and because they specialize in Drupal, many of these professional services companies contribute frequently and are a huge part of our community. Examples are Chapter Three (shown on graph) and Lullabot (shown on graph).
Digital marketing agencies Larger full-service agencies that have marketing-led practices using a variety of tools, typically including Drupal, Adobe Experience Manager, Sitecore, WordPress, etc. They tend to be larger, with the larger agencies employing thousands of people. Examples are Wunderman and Mirum.
System integrators Larger companies that specialize in bringing together different technologies into one solution. Example system agencies are Accenture, TATA Consultancy Services, Capgemini and CI&T.
Technology and infrastructure companies Examples are Acquia (shown on graph), Lingotek, BlackMesh, Rackspace, Pantheon and Platform.sh.
End-users Examples are Pfizer (shown on graph) or NBCUniversal.

A few observations:

  • Almost all of the sponsors in the top 30 are traditional Drupal businesses. Companies like MD Systems (12 employees), Valuebound (34 employees), Chapter Three (27 employees), Commerce Guys (7 employees) and PreviousNext (20 employees) are, despite their size, critical to Drupal's success.

    It's worth highlighting MD Systems, which ranks second in the list of the top 30 contributing organizations, and is the number-one contributor among traditional Drupal businesses. What distinguishes MD Systems from most others is that it has embedded contribution into its corporate philosophy. For every commercial project, MD Systems invests 20% of that project's value back into Drupal. They believe that using commercial projects as the foundation for community contribution leads to more meaningful and healthier contributions for Drupal and a lower total cost of ownership for their customers. This is different from other organizations, where employees are allotted a number of hours per month to contribute outside of customer-facing projects. There is no denying that MD Systems has had a tremendous impact on the Drupal community with contributions that are both frequent and impactful.

  • Compared to these traditional Drupal businesses, Acquia has nearly 800 employees and several full-time Drupal contributors. Acquia's Office of the CTO (OCTO) works to resolve some of the most complex issues on Drupal.org, many of which are not recognized by the credit system (e.g. release management, communication, sprint organizing, and project coordination). However, I believe that Acquia should contribute even more due to our comparative size.
  • No digital marketing agencies show up in the top 30, though some of them are starting to contribute. It is exciting that an increasing number of digital marketing agencies are delivering beautiful experiences using Drupal. As a community, we need to work to ensure that each of these firms are contributing back to the project with the same commitment that we see from firms like Chapter Three, MD Systems or CI&T.
  • The only system integrator in the top 30 is CI&T, which ranked 6th with 664 credits. As far as system integrators are concerned, CI&T is a smaller player with approximately 2,500 employees. However, we do see various system integrators outside of the top 30, including Globant, Capgemini, Sapient and TATA Consultancy Services. Each of these system integrators reported 30 to 70 credits in the past year. Finally, Wipro began contributing this year with 2 credits. We expect, or hope, to see system integrators contribute more and more, especially given the number of Drupal developers they employ. Many have sizable Drupal practices with hundreds of Drupal developers, yet contributing to open source is relatively new and often not well-understood.
  • Infrastructure and software companies play an important role in our community, yet only Acquia appears in the top 30. While Acquia has a professional services division, 75% of the contributions come from the product organization (including the Office of the CTO and the Acquia Lightning team). Other contributing infrastructure companies include Pantheon and Platform.sh, which are both venture-backed platform-as-a-service companies that originated from the Drupal community. Pantheon has 17 credits and Platform.sh has 47 credits. Amazee Labs, who is building an infrastructure business, reported 51 credits. Rackspace is a public company hosting thousands of Drupal sites; they have 48 credits. Lingotek offers cloud-based translation management software and has 94 credits.
  • We saw two end-users in the top 30 corporate sponsors: Pfizer (251 credits, up from 158 credits the year before) and the German company bio.logis (212 credits). Other notable customers outside of the top 30 were Workday, Wolters Kluwer, Burda Media, University of Colorado Boulder, YMCA and OpenY, CARD.com and NBCUniversal.
Contributions by technology companiesSponsored code contributions to Drupal.org from technology and infrastructure companies. The chart does not reflect sponsored code contributions on GitHub, Drupal event sponsorship, and the many forms of value that these companies add to Drupal and other open-source communities.

We can conclude that technology and infrastructure companies, digital marketing agencies, system integrators and end-users are not meaningfully contributing code to Drupal.org today. How can we explain this disparity in comparison to traditional Drupal businesses who contribute the most? We believe the biggest reasons are:

  1. Drupal's strategic importance. A variety of the traditional Drupal agencies have been involved with Drupal for 10 years and almost entirely depend on Drupal to support their business. Given both their expertise and dependence on Drupal, they are most likely to look after Drupal's development and well-being. These organizations are typically recognized as Drupal experts and are sought out by organizations that want to build a Drupal website. Contrast this with most of the digital marketing agencies and system integrators who are sized to work with a diversified portfolio of content management platforms and who are historically only getting started with Drupal and open source. They deliver digital marketing solutions and aren't necessarily sought out for their Drupal expertise. As their Drupal practices grow in size and importance, this could change. In fact, contributing to Drupal can help grow their Drupal business because it helps their name stand out as Drupal experts and gives them a competitive edge with their customers.
  2. The level of experience with Drupal and open source. Drupal aside, many organizations have little or no experience with open source, so it is important that we motivate and teach them to contribute.
  3. Legal reservations. We recognize that some organizations are not legally permitted to contribute, let alone attribute their customers. We hope that will change as open source continues to get adopted.
  4. Tools and process barriers. Drupal contribution still involves a patch-based workflow on Drupal.org's unique issue queue system. This presents a fairly steep learning curve to most developers, who primarily work with more modern and common tools such as GitHub. Getting the code change proposal uploaded is just the first step; getting code changes accepted into an upstream Drupal project — especially Drupal core — is hard work. Peer reviews, gates such as automated testing and documentation, required sign-offs from maintainers and committers, knowledge of best practices and other community norms are a few of the challenges a contributor must face to get code accepted into Drupal.

Consequently, this data shows that the Drupal community can do more to entice companies to contribute code to Drupal.org. The Drupal community has a long tradition of encouraging organizations to share code rather than keep it behind firewalls. While the spirit of the Drupal project cannot be reduced to any single ideology — not every organization can or will share their code — we would like to see organizations continue to prioritize collaboration over individual ownership. Our aim is not to criticize those who do not contribute, but rather to help foster an environment worthy of contribution. Given the vast amount of Drupal users, we believe continuing to encourage organizations and end-users to contribute could be a big opportunity.

There are substantial benefits and business drivers for organizations that contribute: (1) it improves their ability to sell and win deals and (2) it improves their ability to hire. Companies that contribute to Drupal tend to promote their contributions in RFPs and sales pitches. Contributing to Drupal also results in being recognized as a great place to work for Drupal experts.

The uneasy alliance with corporate contributions

As mentioned above, when community-driven open-source projects grow, there is a bigger need for organizations to help drive their development. It almost always creates an uneasy alliance between volunteers and corporations.

This theory played out in the Linux community well before it played out in the Drupal community. The Linux project is 25 years old and has seen a steady increase in the number of corporate contributors for roughly 20 years. While Linux companies like Red Hat and SUSE rank high on the contribution list, so do non-Linux-centric companies such as Samsung, Intel, Oracle and Google. All of these corporate contributors are (or were) using Linux as an integral part of their business.

The 889 organizations that contribute to Drupal (which includes corporations) is more than four times the number of organizations that sponsor development of the Linux kernel. This is significant because Linux is considered "one of the largest cooperative software projects ever attempted". In fairness, Linux has a different ecosystem than Drupal. The Linux business ecosystem has various large organizations (Red Hat, Google, Intel, IBM and SUSE) for whom Linux is very strategic. As a result, many of them employ dozens of full-time Linux contributors and invest millions of dollars in Linux each year.

What projects have sponsors?

In total, the Drupal community worked on 3,183 different projects (modules, themes and distributions) in the 12-month period between July 1, 2016 to June 30, 2017. To understand where the organizations sponsoring Drupal put their money, I've listed the top 20 most sponsored projects:

RankProject nameIssues
1Drupal core4745
2Drupal Commerce (distribution)526
3Webform361
4Open Y (distribution)324
5Paragraphs231
6Inmail223
7User guide218
8JSON API204
9Paragraphs collection200
10Entity browser196
11Diff190
12Group170
13Metatag157
14Facets155
15Commerce Point of Sale (PoS)147
16Search API143
17Open Social (distribution)133
18Drupal voor Gemeenten (distribution)131
19Solr Search122
20Geolocation field118

Who is sponsoring the top 30 contributors?

Rank Username Issues Volunteer Sponsored Not specified Sponsors
1 jrockowitz 537 88% 45% 9% The Big Blue House (239), Kennesaw State University (6), Memorial Sloan Kettering Cancer Center (4)
2 dawehner 421 67% 83% 5% Chapter Three (328), Tag1 Consulting (19), Drupal Association (12), Acquia (5), Comm-press (1)
3 RenatoG 408 0% 100% 0% CI&T (408)
4 bojanz 351 0% 95% 5% Commerce Guys (335), Adapt A/S (38), Bluespark (2)
5 Berdir 335 0% 93% 7% MD Systems (310), Acquia (7)
6 mglaman 334 3% 97% 1% Commerce Guys (319), Thinkbean, LLC (48), LivePerson, Inc (46), Bluespark (22), Universal Music Group (16), Gaggle.net, Inc. (3), Bluehorn Digital (1)
7 Wim Leers 332 14% 87% 2% Acquia (290)
8 alexpott 329 7% 99% 1% Chapter Three (326), TES Global (1)
9 DamienMcKenna 245 2% 95% 4% Mediacurrent (232)
10 jhodgdon 242 0% 1% 99% Drupal Association (2), Poplar ProductivityWare (2)
11 drunken monkey 238 95% 11% 1% Acquia (17), Vizala (8), Wunder Group (1), Sunlime IT Services GmbH (1)
12 naveenvalecha 196 74% 55% 1% Acquia (152), Google Summer of Code (7), QED42 (1)
13 Munavijayalakshmi 192 0% 100% 0% Valuebound (192)
14 borisson_ 191 66% 39% 22% Dazzle (70), Acquia (6)
15 yongt9412 189 0% 97% 3% MD Systems (183), Acquia (6)
16 klausi 185 9% 61% 32% epiqo (112)
17 Sam152 184 59% 92% 7% PreviousNext (168), amaysim Australia Ltd. (5), Code Drop (2)
18 miro_dietiker 182 0% 99% 1% MD Systems (181)
19 Pavan B S 180 0% 98% 2% Valuebound (177)
20 ajay_reddy 176 100% 99% 0% Valuebound (180), Drupal Bangalore Community (154)
21 phenaproxima 172 0% 99% 1% Acquia (170)
22 sanchiz 162 0% 99% 1% Drupal Ukraine Community (107), Vinzon (101), FFW (60), Open Y (52)
23 slashrsm 161 6% 95% 3% MD Systems (153), Acquia (47)
24 jhedstrom 155 4% 92% 4% Phase2 (143), Workday, Inc. (134), Memorial Sloan Kettering Cancer Center (1)
25 xjm 151 0% 91% 9% Acquia (137)
26 catch 147 3% 83% 16% Third and Grove (116), Tag1 Consulting (6)
27 larowlan 145 12% 92% 7% PreviousNext (133), University of Technology, Sydney (30), amaysim Australia Ltd. (6), Australian Competition and Consumer Commission (ACCC) (1), Department of Justice & Regulation, Victoria (1)
28 rakesh.gectcr 141 100% 91% 0% Valuebound (128)
29 benjy 139 0% 94% 6% PreviousNext (129), Brisbane City Council (8), Code Drop (1)
30 dhruveshdtripathi 138 15% 100% 0% DevsAdda (138), OpenSense Labs (44)

We observe that the top 30 contributors are sponsored by 46 organizations. This kind of diversity is aligned with our desire not to see Drupal controlled by a single organization. These top contributors and organizations are from many different parts of the world and work with customers large and small. Nonetheless, we will continue to benefit from more diversity.

Evolving the credit system

Like Drupal itself, the credit system on Drupal.org is an evolving tool. Ultimately, the credit system will only be useful when the community uses it, understands its shortcomings, and suggests constructive improvements. In highlighting the organizations that sponsor the development of code on Drupal.org, we hope to elicit responses that help evolve the credit system into something that incentivizes business to sponsor more work and enables more people to participate in our community, learn from others, teach newcomers and make positive contributions. Drupal is a positive force for change and we wish to use the credit system to highlight (at least some of) the work of our diverse community, which includes volunteers, companies, nonprofits, governments, schools, universities, individuals, and other groups.

One of the challenges with the existing credit system is it has no way of "weighting" contributions. A typo fix counts just as much as giving multiple detailed technical reviews on a critical core issue. This appears to have the effect of incentivizing organizations' employees to work on "lower-hanging fruit issues", because this bumps their companies' names in the rankings. One way to help address this might be to adjust the credit ranking algorithm to consider things such as issue priority, patch size, and so on. This could help incentivize companies to work on larger and more important problems and save coding standards improvements for new contributor sprints. Implementing a scoring system that ranks the complexity of an issue would also allow us to develop more accurate reports of contributed work.

Conclusion

Our data confirms Drupal is a vibrant community full of contributors who are constantly evolving and improving the software. While we have amazing geographic diversity, we need greater gender diversity. Our analysis of the Drupal.org credit data concludes that most contributions to Drupal are sponsored. At the same time, the data shows that volunteer contribution remains very important to Drupal.

As a community, we need to understand that a healthy open-source ecosystem includes more than traditional Drupal businesses that contribute the most. For example, we don't see a lot of contribution from the larger digital marketing agencies, system integrators, technology companies, or end-users of Drupal — we believe that might come as these organizations build out their Drupal practices and Drupal becomes more strategic for them.

To grow and sustain Drupal, we should support those that contribute to Drupal and find ways to get those that are not contributing involved in our community. We invite you to help us continue to strengthen our ecosystem.

Special thanks to Tim Lehnen and Neil Drumm from the Drupal Association for providing us with the Drupal.org credit system data and for supporting us during our research. I would also like to extend a special thanks to Matthew Tift for helping to lay the foundation for this research, collaborating on last year's blog post, and for reviewing this year's edition. Finally, thanks to Angie Byron, Gábor Hojtsy, Jess (xjm), Preston So, Ted Bowman, Wim Leers and Gigi Anderson for providing feedback during the writing process.

Last year, Matthew Tift and I examined Drupal.org's commit data to understand who develops Drupal, how much of that work is sponsored, and where that sponsorship comes from. We published our analysis in a blog post called "Who Sponsors Drupal Development?". A year later, I wanted to present an update. This year's report will also cover additional data, including gender and geographical diversity, and project sponsorship.

Understanding how an open-source project works is important because it establishes a benchmark for project health and scalability. Scaling an open-source project is a difficult task. As an open-source project's rate of adoption grows, the number of people that benefit from the project also increases. Often the open-source project also becomes more complex as it expands, which means that the economic reward of helping to improve the project decreases.

A recent article on the Bitcoin and Ethereum contributor communities illustrates this disparity perfectly. Ethereum and Bitcoin have market capitalizations valued at $30 billion and $70 billion, respectively. However, both projects have fewer than 40 meaningful contributors, and contribution isn't growing despite the rising popularity of cryptocurrency.

Number of Bitcoin contributors between 2010 and 2017According to Bitcoin's GitHub data, Bitcoin has less than 40 active contributors.
Number of Ethereum contributors between 2014 and 2017According to Ethereum's GitHub data, Ethereum has less than 20 active contributors.

Drupal, by comparison, has a diverse community of contributors. In the 12-month period between July 1, 2016 to June 30, 2017 we saw code contributions on Drupal.org from 7,240 different individuals and 889 different companies. This does not mean that Drupal is exempt from the challenges of scaling an open-source project. We hope that this report provides transparency about Drupal project development and encourages more individuals and organizations incentive to contribute. We also will highlight areas where our community can and should do better.

What is the Drupal.org credit system?

In the spring of 2015, after proposing ideas for giving credit and discussing various approaches at length, Drupal.org added the ability for people to attribute their work to an organization or customer in the Drupal.org issue queues. Maintainers of Drupal modules, themes and distributions can award issues credits to people who help resolve issues with code, translations, documentation, design and more.

Example issue credit on drupal orgA screenshot of an issue comment on Drupal.org. You can see that jamadar worked on this patch as a volunteer, but also as part of his day job working for TATA Consultancy Services on behalf of their customer, Pfizer.

Credits are a powerful motivator for both individuals and organizations. Accumulating credits provides individuals with a way to showcase their expertise. Organizations can utilize credits to help recruit developers or to increase their visibility in the Drupal.org marketplace.

While the benefits are evident, it is important to note a few of the limitations in Drupal.org's current credit system:

  • Contributing to issues on Drupal.org is not the only way to contribute. Other activities, such as sponsoring events, promoting Drupal, and providing help and mentorship, are important to the long-term health of the Drupal project. Many of these activities are not currently captured by the credit system. For this post, we chose to only look at code contributions.
  • We acknowledge that parts of Drupal are developed on GitHub and therefore aren't fully credited on Drupal.org. The actual number of contributions and contributors could be significantly higher than what we report.
  • Even when development is done on Drupal.org, the credit system is not used consistently; because using the credit system is optional, a lot of code committed on Drupal.org has no or incomplete contribution credits.
  • Not all code credits are the same. We currently don't have a way to account for the complexity and quality of contributions; one person might have worked several weeks for just one credit, while another person might receive a credit for ten minutes of work. In the future, we should consider issuing credit data in conjunction with issue priority, patch size, etc. We can also reduce the need for trivial credits by automating patch rerolls and automating coding style fixes.

Who is working on Drupal?

For our analysis we looked at all the issues that were marked "closed" or "fixed" in the 12-month period from July 1, 2016 to June 30, 2017. What we learned is that there were 23,238 issues marked "closed" or "fixed", a 22% increase from the 19,095 issues in the 2015-2016 period. Those 23,238 issues had 42,449 issue credits, a 30% increase from the 32,711 issue credits recorded in the previous year. Issue credits against Drupal core remained roughly the same year over year, meaning almost all of this growth came from increased activity in contributed projects. This is no surprise. Drupal development is cyclical, and during this period of the Drupal 8 development cycle, most of the Drupal community has been focused on porting modules from Drupal 7 to Drupal 8. Of the 42,449 issue credits reported this year, 20% (8,619 credits) were for Drupal core, while 80% (33,830 credits) went to contributed themes, modules and distributions.

Compared to the previous year, we also saw an increase in both the number of people contributing and the number of organizations contributing. Drupal.org received code contributions from 7,240 different individuals and 889 different organizations.

Contributions by individuals vs organizationsThe number of individual contributors is up 28% year over year and the number of organizations contributing is up 26% year over year.

While the number of individual contributors rose, a relatively small number of individuals still do the majority of the work. Approximately 47% of individual contributors received just one credit. Meanwhile, the top 30 contributors (the top 0.4%) account for over 17% of the total credits, indicating that these individuals put an incredible amount of time and effort in developing Drupal and its contributed projects:

RankUsernameIssues
1jrockowitz537
2dawehner421
3RenatoG408
4bojanz351
5Berdir335
6mglaman334
7Wim Leers332
8alexpott329
9DamienMcKenna245
10jhodgdon242
11drunken monkey238
12naveenvalecha196
13Munavijayalakshmi192
14borisson_191
15yongt9412189
16klausi185
17Sam152184
18miro_dietiker182
19Pavan B S180
20ajay_reddy176
21phenaproxima172
22sanchiz162
23slashrsm161
24jhedstrom155
25xjm151
26catch147
27larowlan145
28rakesh.gectcr141
29benjy139
30dhruveshdtripathi138

Out of the top 30 contributors featured, 19 were also recognized as top contributors in our 2015-2016 report. These Drupalists' dedication and continued contribution to the project has been crucial to Drupal's development. It's also exciting to see 11 new names on the list. This mobility is a testament to the community's evolution and growth.

Next, we looked at both the gender and geographic diversity of Drupal.org code contributors. While these are only two examples of diversity, this is the only available data that contributors can choose to share on their Drupal.org profiles. The reported data shows that only 6% of the recorded contributions were made by contributors that identify as female, which indicates a steep gender gap. Like in most open-source projects, the gender imbalance in Drupal is profound and underscores the need to continue fostering diversity and inclusion in our community.

Contributions by genderThe gender representation behind the issue credits. Only 6% of the recorded contributions are by women.
When measuring geographic diversity, we saw individual contributors from 6 different continents and 116 different countries:
Contributions by continent
Contributions by countryThe top 20 countries from which contributions originate. The data is compiled by aggregating the countries of all individual contributors behind each commit. Note that the geographical location of contributors doesn't always correspond with the origin of their sponsorship. Wim Leers, for example, works from Belgium, but his funding comes from Acquia, which has the majority of its customers in North America.

How much of the work is sponsored?

Drupal is used by more than one million websites. The vast majority of the individuals and organizations behind these Drupal websites never participate in the development of the project. They might use the software as it is or might not feel the need to help drive its development. We have to provide more incentive for these individuals and organizations to contribute back to the project.

Issue credits can be marked as "volunteer" and "sponsored" simultaneously (shown in jamadar's screenshot near the top of this post). This could be the case when a contributor does the minimum required work to satisfy the customer's need, in addition to using their spare time to add extra functionality.

While Drupal started out as a 100% volunteer-driven project, today the majority of the code on Drupal.org is sponsored by organizations. Only 11% of the commit credits that we examined in 2016-2017 were "purely volunteer" credits (4,498 credits), in stark contrast to the 46% that were "purely sponsored". In other words, there were four times as many "purely sponsored" credits as "purely volunteer" credits.

A few comparisons with the 2015-2016 data:

  • The credit system is used more. Between July 1, 2015 and June 30, 2016, 37% of all credits had no attribution while in the more recent period between July 1, 2016 to June 30, 2017, only 28% of credits lacked attribution. More people have become aware of the credit system, the attribution options, and their benefits.
  • Sponsored credits are growing faster than volunteer credits. Both "purely volunteer" and "purely sponsored" credits grew, but "purely sponsored" credits grew faster. There are two reasons why this could be the case: (1) more contributions are sponsored and (2) organizations are more likely to use the credit system compared to volunteers.
Contributions by volunteer vs sponsored

No data is perfect, but it feels safe to conclude that most of the work on Drupal is sponsored. At the same time, the data shows that volunteer contribution remains very important to Drupal. Maybe most importantly, while the number of volunteers and sponsors has grown year over year in absolute terms, sponsored contributions appear to be growing faster than volunteer contributions. This is consistent with how open source projects grow and scale.

Who is sponsoring the work?

Now that we have established that most of the work on Drupal is sponsored, we want to study which organizations contribute to Drupal. While 889 different organizations contributed to Drupal, approximately 50% of them received four credits or fewer. The top 30 organizations (roughly the top 3%) account for about 48% of the total credits, which implies that the top 30 companies play a crucial role in the health of the Drupal project. The graph below shows the top 30 organizations and the number of credits they received between July 1, 2016 and June 30, 2017:

Top 30 organizations contributing to DrupalThe top 30 contributing organizations based on the number of Drupal.org commit credits.

While not immediately obvious from the graph above, different types of companies are active in Drupal's ecosystem:

Category Description
Traditional Drupal businesses Small-to-medium-sized professional services companies that make money primarily using Drupal. They typically employ fewer than 100 employees, and because they specialize in Drupal, many of these professional services companies contribute frequently and are a huge part of our community. Examples are Chapter Three (shown on graph) and Lullabot (shown on graph).
Digital marketing agencies Larger full-service agencies that have marketing-led practices using a variety of tools, typically including Drupal, Adobe Experience Manager, Sitecore, WordPress, etc. They tend to be larger, with the larger agencies employing thousands of people. Examples are Wunderman and Mirum.
System integrators Larger companies that specialize in bringing together different technologies into one solution. Example system agencies are Accenture, TATA Consultancy Services, Capgemini and CI&T.
Technology and infrastructure companies Examples are Acquia (shown on graph), Lingotek, BlackMesh, Rackspace, Pantheon and Platform.sh.
End-users Examples are Pfizer (shown on graph) or NBCUniversal.

A few observations:

  • Almost all of the sponsors in the top 30 are traditional Drupal businesses. Companies like MD Systems (12 employees), Valuebound (34 employees), Chapter Three (27 employees), Commerce Guys (7 employees) and PreviousNext (20 employees) are, despite their size, critical to Drupal's success.

    It's worth highlighting MD Systems, which ranks second in the list of the top 30 contributing organizations, and is the number-one contributor among traditional Drupal businesses. What distinguishes MD Systems from most others is that it has embedded contribution into its corporate philosophy. For every commercial project, MD Systems invests 20% of that project's value back into Drupal. They believe that using commercial projects as the foundation for community contribution leads to more meaningful and healthier contributions for Drupal and a lower total cost of ownership for their customers. This is different from other organizations, where employees are allotted a number of hours per month to contribute outside of customer-facing projects. There is no denying that MD Systems has had a tremendous impact on the Drupal community with contributions that are both frequent and impactful.

  • Compared to these traditional Drupal businesses, Acquia has nearly 800 employees and several full-time Drupal contributors. Acquia's Office of the CTO (OCTO) works to resolve some of the most complex issues on Drupal.org, many of which are not recognized by the credit system (e.g. release management, communication, sprint organizing, and project coordination). However, I believe that Acquia should contribute even more due to our comparative size.
  • No digital marketing agencies show up in the top 30, though some of them are starting to contribute. It is exciting that an increasing number of digital marketing agencies are delivering beautiful experiences using Drupal. As a community, we need to work to ensure that each of these firms are contributing back to the project with the same commitment that we see from firms like Chapter Three, MD Systems or CI&T.
  • The only system integrator in the top 30 is CI&T, which ranked 6th with 664 credits. As far as system integrators are concerned, CI&T is a smaller player with approximately 2,500 employees. However, we do see various system integrators outside of the top 30, including Globant, Capgemini, Sapient and TATA Consultancy Services. Each of these system integrators reported 30 to 70 credits in the past year. Finally, Wipro began contributing this year with 2 credits. We expect, or hope, to see system integrators contribute more and more, especially given the number of Drupal developers they employ. Many have sizable Drupal practices with hundreds of Drupal developers, yet contributing to open source is relatively new and often not well-understood.
  • Infrastructure and software companies play an important role in our community, yet only Acquia appears in the top 30. While Acquia has a professional services division, 75% of the contributions come from the product organization (including the Office of the CTO and the Acquia Lightning team). Other contributing infrastructure companies include Pantheon and Platform.sh, which are both venture-backed platform-as-a-service companies that originated from the Drupal community. Pantheon has 17 credits and Platform.sh has 47 credits. Amazee Labs, who is building an infrastructure business, reported 51 credits. Rackspace is a public company hosting thousands of Drupal sites; they have 48 credits. Lingotek offers cloud-based translation management software and has 94 credits.
  • We saw two end-users in the top 30 corporate sponsors: Pfizer (251 credits, up from 158 credits the year before) and the German company bio.logis (212 credits). Other notable customers outside of the top 30 were Workday, Wolters Kluwer, Burda Media, University of Colorado Boulder, YMCA and OpenY, CARD.com and NBCUniversal.
Contributions by technology companiesSponsored code contributions to Drupal.org from technology and infrastructure companies. The chart does not reflect sponsored code contributions on GitHub, Drupal event sponsorship, and the many forms of value that these companies add to Drupal and other open-source communities.

We can conclude that technology and infrastructure companies, digital marketing agencies, system integrators and end-users are not meaningfully contributing code to Drupal.org today. How can we explain this disparity in comparison to traditional Drupal businesses who contribute the most? We believe the biggest reasons are:

  1. Drupal's strategic importance. A variety of the traditional Drupal agencies have been involved with Drupal for 10 years and almost entirely depend on Drupal to support their business. Given both their expertise and dependence on Drupal, they are most likely to look after Drupal's development and well-being. These organizations are typically recognized as Drupal experts and are sought out by organizations that want to build a Drupal website. Contrast this with most of the digital marketing agencies and system integrators who are sized to work with a diversified portfolio of content management platforms and who are historically only getting started with Drupal and open source. They deliver digital marketing solutions and aren't necessarily sought out for their Drupal expertise. As their Drupal practices grow in size and importance, this could change. In fact, contributing to Drupal can help grow their Drupal business because it helps their name stand out as Drupal experts and gives them a competitive edge with their customers.
  2. The level of experience with Drupal and open source. Drupal aside, many organizations have little or no experience with open source, so it is important that we motivate and teach them to contribute.
  3. Legal reservations. We recognize that some organizations are not legally permitted to contribute, let alone attribute their customers. We hope that will change as open source continues to get adopted.
  4. Tools and process barriers. Drupal contribution still involves a patch-based workflow on Drupal.org's unique issue queue system. This presents a fairly steep learning curve to most developers, who primarily work with more modern and common tools such as GitHub. Getting the code change proposal uploaded is just the first step; getting code changes accepted into an upstream Drupal project — especially Drupal core — is hard work. Peer reviews, gates such as automated testing and documentation, required sign-offs from maintainers and committers, knowledge of best practices and other community norms are a few of the challenges a contributor must face to get code accepted into Drupal.

Consequently, this data shows that the Drupal community can do more to entice companies to contribute code to Drupal.org. The Drupal community has a long tradition of encouraging organizations to share code rather than keep it behind firewalls. While the spirit of the Drupal project cannot be reduced to any single ideology — not every organization can or will share their code — we would like to see organizations continue to prioritize collaboration over individual ownership. Our aim is not to criticize those who do not contribute, but rather to help foster an environment worthy of contribution. Given the vast amount of Drupal users, we believe continuing to encourage organizations and end-users to contribute could be a big opportunity.

There are substantial benefits and business drivers for organizations that contribute: (1) it improves their ability to sell and win deals and (2) it improves their ability to hire. Companies that contribute to Drupal tend to promote their contributions in RFPs and sales pitches. Contributing to Drupal also results in being recognized as a great place to work for Drupal experts.

The uneasy alliance with corporate contributions

As mentioned above, when community-driven open-source projects grow, there is a bigger need for organizations to help drive their development. It almost always creates an uneasy alliance between volunteers and corporations.

This theory played out in the Linux community well before it played out in the Drupal community. The Linux project is 25 years old and has seen a steady increase in the number of corporate contributors for roughly 20 years. While Linux companies like Red Hat and SUSE rank high on the contribution list, so do non-Linux-centric companies such as Samsung, Intel, Oracle and Google. All of these corporate contributors are (or were) using Linux as an integral part of their business.

The 889 organizations that contribute to Drupal (which includes corporations) is more than four times the number of organizations that sponsor development of the Linux kernel. This is significant because Linux is considered "one of the largest cooperative software projects ever attempted". In fairness, Linux has a different ecosystem than Drupal. The Linux business ecosystem has various large organizations (Red Hat, Google, Intel, IBM and SUSE) for whom Linux is very strategic. As a result, many of them employ dozens of full-time Linux contributors and invest millions of dollars in Linux each year.

What projects have sponsors?

In total, the Drupal community worked on 3,183 different projects (modules, themes and distributions) in the 12-month period between July 1, 2016 to June 30, 2017. To understand where the organizations sponsoring Drupal put their money, I've listed the top 20 most sponsored projects:

RankProject nameIssues
1Drupal core4745
2Drupal Commerce (distribution)526
3Webform361
4Open Y (distribution)324
5Paragraphs231
6Inmail223
7User guide218
8JSON API204
9Paragraphs collection200
10Entity browser196
11Diff190
12Group170
13Metatag157
14Facets155
15Commerce Point of Sale (PoS)147
16Search API143
17Open Social (distribution)133
18Drupal voor Gemeenten (distribution)131
19Solr Search122
20Geolocation field118

Who is sponsoring the top 30 contributors?

Rank Username Issues Volunteer Sponsored Not specified Sponsors
1 jrockowitz 537 88% 45% 9% The Big Blue House (239), Kennesaw State University (6), Memorial Sloan Kettering Cancer Center (4)
2 dawehner 421 67% 83% 5% Chapter Three (328), Tag1 Consulting (19), Drupal Association (12), Acquia (5), Comm-press (1)
3 RenatoG 408 0% 100% 0% CI&T (408)
4 bojanz 351 0% 95% 5% Commerce Guys (335), Adapt A/S (38), Bluespark (2)
5 Berdir 335 0% 93% 7% MD Systems (310), Acquia (7)
6 mglaman 334 3% 97% 1% Commerce Guys (319), Thinkbean, LLC (48), LivePerson, Inc (46), Bluespark (22), Universal Music Group (16), Gaggle.net, Inc. (3), Bluehorn Digital (1)
7 Wim Leers 332 14% 87% 2% Acquia (290)
8 alexpott 329 7% 99% 1% Chapter Three (326), TES Global (1)
9 DamienMcKenna 245 2% 95% 4% Mediacurrent (232)
10 jhodgdon 242 0% 1% 99% Drupal Association (2), Poplar ProductivityWare (2)
11 drunken monkey 238 95% 11% 1% Acquia (17), Vizala (8), Wunder Group (1), Sunlime IT Services GmbH (1)
12 naveenvalecha 196 74% 55% 1% Acquia (152), Google Summer of Code (7), QED42 (1)
13 Munavijayalakshmi 192 0% 100% 0% Valuebound (192)
14 borisson_ 191 66% 39% 22% Dazzle (70), Acquia (6)
15 yongt9412 189 0% 97% 3% MD Systems (183), Acquia (6)
16 klausi 185 9% 61% 32% epiqo (112)
17 Sam152 184 59% 92% 7% PreviousNext (168), amaysim Australia Ltd. (5), Code Drop (2)
18 miro_dietiker 182 0% 99% 1% MD Systems (181)
19 Pavan B S 180 0% 98% 2% Valuebound (177)
20 ajay_reddy 176 100% 99% 0% Valuebound (180), Drupal Bangalore Community (154)
21 phenaproxima 172 0% 99% 1% Acquia (170)
22 sanchiz 162 0% 99% 1% Drupal Ukraine Community (107), Vinzon (101), FFW (60), Open Y (52)
23 slashrsm 161 6% 95% 3% MD Systems (153), Acquia (47)
24 jhedstrom 155 4% 92% 4% Phase2 (143), Workday, Inc. (134), Memorial Sloan Kettering Cancer Center (1)
25 xjm 151 0% 91% 9% Acquia (137)
26 catch 147 3% 83% 16% Third and Grove (116), Tag1 Consulting (6)
27 larowlan 145 12% 92% 7% PreviousNext (133), University of Technology, Sydney (30), amaysim Australia Ltd. (6), Australian Competition and Consumer Commission (ACCC) (1), Department of Justice & Regulation, Victoria (1)
28 rakesh.gectcr 141 100% 91% 0% Valuebound (128)
29 benjy 139 0% 94% 6% PreviousNext (129), Brisbane City Council (8), Code Drop (1)
30 dhruveshdtripathi 138 15% 100% 0% DevsAdda (138), OpenSense Labs (44)

We observe that the top 30 contributors are sponsored by 46 organizations. This kind of diversity is aligned with our desire not to see Drupal controlled by a single organization. These top contributors and organizations are from many different parts of the world and work with customers large and small. Nonetheless, we will continue to benefit from more diversity.

Evolving the credit system

Like Drupal itself, the credit system on Drupal.org is an evolving tool. Ultimately, the credit system will only be useful when the community uses it, understands its shortcomings, and suggests constructive improvements. In highlighting the organizations that sponsor the development of code on Drupal.org, we hope to elicit responses that help evolve the credit system into something that incentivizes business to sponsor more work and enables more people to participate in our community, learn from others, teach newcomers and make positive contributions. Drupal is a positive force for change and we wish to use the credit system to highlight (at least some of) the work of our diverse community, which includes volunteers, companies, nonprofits, governments, schools, universities, individuals, and other groups.

One of the challenges with the existing credit system is it has no way of "weighting" contributions. A typo fix counts just as much as giving multiple detailed technical reviews on a critical core issue. This appears to have the effect of incentivizing organizations' employees to work on "lower-hanging fruit issues", because this bumps their companies' names in the rankings. One way to help address this might be to adjust the credit ranking algorithm to consider things such as issue priority, patch size, and so on. This could help incentivize companies to work on larger and more important problems and save coding standards improvements for new contributor sprints. Implementing a scoring system that ranks the complexity of an issue would also allow us to develop more accurate reports of contributed work.

Conclusion

Our data confirms Drupal is a vibrant community full of contributors who are constantly evolving and improving the software. While we have amazing geographic diversity, we need greater gender diversity. Our analysis of the Drupal.org credit data concludes that most contributions to Drupal are sponsored. At the same time, the data shows that volunteer contribution remains very important to Drupal.

As a community, we need to understand that a healthy open-source ecosystem includes more than traditional Drupal businesses that contribute the most. For example, we don't see a lot of contribution from the larger digital marketing agencies, system integrators, technology companies, or end-users of Drupal — we believe that might come as these organizations build out their Drupal practices and Drupal becomes more strategic for them.

To grow and sustain Drupal, we should support those that contribute to Drupal and find ways to get those that are not contributing involved in our community. We invite you to help us continue to strengthen our ecosystem.

Special thanks to Tim Lehnen and Neil Drumm from the Drupal Association for providing us with the Drupal.org credit system data and for supporting us during our research. I would also like to extend a special thanks to Matthew Tift for helping to lay the foundation for this research, collaborating on last year's blog post, and for reviewing this year's edition. Finally, thanks to Angie Byron, Gábor Hojtsy, Jess (xjm), Preston So, Ted Bowman, Wim Leers and Gigi Anderson for providing feedback during the writing process.

September 11, 2017

In order to further secure access to my workstation, after the switch to Gentoo sources, I now enabled two-factor authentication through my Yubico U2F USB device. Well, at least for local access - remote access through SSH requires both userid/password as well as the correct SSH key, by chaining authentication methods in OpenSSH.

Enabling U2F on (Gentoo) Linux is fairly easy. The various guides online which talk about the pam_u2f setup are indeed correct that it is fairly simple. For completeness sake, I've documented what I know on the Gentoo Wiki, as the pam_u2f article.

The setup, basically

The setup of U2F is done in a number of steps: 1. Validate that the kernel is ready for the USB device 2. Install the PAM module and supporting tools 3. Generate the necessary data elements for each user (keys and such) 4. Configure PAM to require authentication through the U2F key

For the kernel, the configuration item needed is the raw HID device support. Now, in current kernels, two settings are available that both talk about raw HID device support: CONFIG_HIDRAW is the general raw HID device support, while CONFIG_USB_HIDDEV is the USB-specific raw HID device support.

It is very well possible that only a single one is needed, but both where active on my kernel configuration already, and Internet sources are not clear which one is needed, so let's assume for now both are.

Next, the PAM module needs to be installed. On Gentoo, this is a matter of installing the pam\_u2f package, as the necessary dependencies will be pulled in automatically:

~# emerge pam_u2f

Next, for each user, a registration has to be made. This registration is needed for the U2F components to be able to correctly authenticate the use of a U2F key for a particular user. This is done with pamu2fcfg:

~$ pamu2fcfg -u<username> > ~/.config/Yubico/u2f_keys

The U2F USB key must be plugged in when the command is executed, as a succesful keypress (on the U2F device) is needed to complete the operation.

Finally, enable the use of the pam\_u2f module in PAM. On my system, this is done through the /etc/pam.d/system-local-login PAM configuration file used by all local logon services.

auth     required     pam_u2f.so

Consider the problems you might face

When fiddling with PAM, it is important to keep in mind what could fail. During the setup, it is recommended to have an open administrative session on the system so that you can validate if the PAM configuration works, without locking yourself out of the system.

But other issues need to be considered as well.

My Yubico U2F USB key might have a high MTBF (Mean Time Between Failures) value, but once it fails, it would lock me out of my workstation (and even remote services and servers that use it). For that reason, I own a second one, safely stored, but is a valid key nonetheless for my workstation and remote systems/services. Given the low cost of a simple U2F key, it is a simple solution for this threat.

Another issue that could come up is a malfunction in the PAM module itself. For me, this is handled by having remote SSH access done without this PAM module (although other PAM modules are still involved, so a generic PAM failure itself wouldn't resolve this). Of course, worst case, the system needs to be rebooted in single user mode.

One issue that I faced was the SELinux policy. Some applications that provide logon services don't have the proper rights to handle U2F, and because PAM just works in the address space (and thus SELinux domain) of the application, the necessary privileges need to be added to these services. My initial investigation revealed the following necessary policy rules (refpolicy-style);

udev_search_pids(...)
udev_read_db(...)
dev_rw_generic_usb_dev(...)

The first two rules are needed because the operation to trigger the USB key uses the udev tables to find out where the key is located/attached, before it interacts with it. This interaction is then controlled through the first rule.

Simple yet effective

Enabling the U2F authentication on the system is very simple, and gives a higher confidence that malicious activities through regular accounts will have it somewhat more challenging to switch to a more privileged session (one control is the SELinux policy of course, but for those domains that are allowed to switch then the PAM-based authentication is another control), as even evesdropping on my password (or extracting it from memory) won't suffice to perform a successful authentication.

If you want to use a different two-factor authentication, check out the use of the Google authenticator, another nice article on the Gentoo wiki. It is also possible to use Yubico keys for remote authentication, but that uses the OTP (One Time Password) functionality which isn't active on the Yubico keys that I own.

September 10, 2017

The post Coming soon: Oh Dear! – Monitoring for the encrypted web appeared first on ma.ttias.be.

I'm excited to announce a new project I'm working on: Oh Dear!

The goal of Oh Dear! is to provide modern monitoring & feedback for sites that run on HTTPS. With Chrome's soon-to-be-released version that marks any input on non-HTTPS pages as "Not Secure", that target audience is huge.

The baseline below I think sums it up very accurately.

Many users only look at their certificate expiration dates when running HTTPS sites and -- hopefully -- renew in time. But that's only a small part of the journey to HTTPS. I've ranted about messing up HTTPS often enough that I don't want to repeat myself anymore.

What does Oh Dear! offer?

From my old rant, the summary from way-back-then still stands today:

Browsers don't care if your HTTPS config is 95% perfect. They'll destroy the visitor's experience if you don't nail it for the full 100%.

There's many things that can go wrong with deploying HTTPS, including;

  • Expired certificates
  • Revoked certificates
  • Missing intermediate certificates in your chain
  • Mixed content on your site
  • Bad or insecure TLS ciphers used in the config
  • Incorrectly configured OCSP stapling
  • Badly pinned keys with HPKP
  • ...

Oh Dear! monitors for each and every one of those things, and more.

Included in Oh Dear! is Certificate Transparency reporting, so you can get notified whenever a new certificate is issued for one of your domains, intentional or otherwise.

Meet the team

Unlike my usual projects, this time I'm working together with smart folks to help make Oh Dear! a success.

The team consists of Dries Vints, Freek Van der Herten & me. We're all active in the Laravel community. Dries & Freek go way back, I only got to know these smart men a little over a year ago.

Join the beta

We're not open to the public yet, but there's a beta program you can subscribe to in order to get access to Oh Dear!.

If you run a website on HTTPS -- and chances are, you do -- don't let a bad certificate or configuration ruin your day. Trust us to monitor it for you and report any errors, before your visitors do.

Go check out our app at ohdearapp.com or follow us on Twitter via @OhDearApp.

The post Coming soon: Oh Dear! – Monitoring for the encrypted web appeared first on ma.ttias.be.

As I threatened to do in my previous post, I'm going to talk about PR 219251 for a bit. The bug report dates from only a few months ago, but the first report (that I can remeber) actually came from Shawn Webb on Twitter, of all places: backtrace

Despite there being a stacktrace it took quite a while (nearly 6 months in fact) before I figured this one out.

It took Reshad Patuck managing to distill the problem down to a small-ish test script to make real progress on this. His testcase meant that I could get core dumps and experiment. It also provided valuable clues because it could be tweaked to see what elements were required to trigger the panic.

This test script starts a (vnet) jail, adds an epair interface to it, sets up pf in the jail, and then reloads the pf rules on the host. Interestingly the panic does not seem to occur if that last step is not included.

Time to take a closer look at the code that breaks:

u_int32_t
pf_state_expires(const struct pf_state *state)
{
        u_int32_t       timeout;
        u_int32_t       start;
        u_int32_t       end;
        u_int32_t       states;

        /* handle all PFTM_* > PFTM_MAX here */
        if (state->timeout == PFTM_PURGE)
                return (time_uptime);
        KASSERT(state->timeout != PFTM_UNLINKED,
            ("pf_state_expires: timeout == PFTM_UNLINKED"));
        KASSERT((state->timeout < PFTM_MAX),
            ("pf_state_expires: timeout > PFTM_MAX"));
        timeout = state->rule.ptr->timeout[state->timeout];
        if (!timeout)
                timeout = V_pf_default_rule.timeout[state->timeout];
        start = state->rule.ptr->timeout[PFTM_ADAPTIVE_START];
        if (start) {
                end = state->rule.ptr->timeout[PFTM_ADAPTIVE_END];
                states = counter_u64_fetch(state->rule.ptr->states_cur);
        } else {
                start = V_pf_default_rule.timeout[PFTM_ADAPTIVE_START];
                end = V_pf_default_rule.timeout[PFTM_ADAPTIVE_END];
                states = V_pf_status.states;
        }
        if (end && states > start && start < end) {
                if (states < end)
                        return (state->expire + timeout * (end - states) /
                            (end - start));
                else
                        return (time_uptime);
        }
        return (state->expire + timeout);
}

Specifically, the following line:

	states = counter_u64_fetch(state->rule.ptr->states_cur);

We try to fetch a counter value here, but instead we dereference a bad pointer. There's two here, so already we need more information. Inspection of the core dump reveals that the state pointer is valid, and contains sane information. The rule pointer (rule.ptr) points to a sensible location, but the data is mostly 0xdeadc0de. This is the memory allocator being helpful (in debug mode) and writing garbage over freed memory, to make use-after-free bugs like this one easier to find.

In other words: the rule has been free()d while there was still a state pointing to it. Somehow we have a state (describing a connection pf knows about) which points to a rule which no longer exists.

The core dump also shows that the problem always occurs with states and rules in the default vnet (i.e. the host pf instance), not one of the pf instances in one of the vnet jails. That matches with the observation that the test script does not trigger the panic unless we also reload the rules on the host.

Great, we know what's wrong, but now we need to work out how we can get into this state. At this point we're going to have to learn something about how rules and states get cleaned up in pf. Don't worry if you had no idea, because before this bug I didn't either.

The states keep a pointer to the rule they match, so when rules are changed (or removed) we can't just delete them. States get cleaned up when connections are closed or they time out. This means we have to keep old rules around until the states that use them expire.

When rules are removed pf_unlink_rule() adds then to the V_pf_unlinked_rules list (more on that funny V_ prefix later). From time to time the pf purge thread will run over all states and mark the rules that are used by a state. Once that's done for all states we know that all rules that are not marked as in-use can be removed (because none of the states use it).

That can be a lot of work if we've got a lot of states, so pf_purge_thread() breaks that up into smaller chuncks, iterating only part of the state table on every run. Let's look at that code:

void
pf_purge_thread(void *unused __unused)
{
        VNET_ITERATOR_DECL(vnet_iter);
        u_int idx = 0;

        sx_xlock(&pf_end_lock);
        while (pf_end_threads == 0) {
                sx_sleep(pf_purge_thread, &pf_end_lock, 0, "pftm", hz / 10);

                VNET_LIST_RLOCK();
                VNET_FOREACH(vnet_iter) {
                        CURVNET_SET(vnet_iter);


                        /* Wait until V_pf_default_rule is initialized. */
                        if (V_pf_vnet_active == 0) {
                                CURVNET_RESTORE();
                                continue;
                        }

                        /*
                         *  Process 1/interval fraction of the state
                         * table every run.
                         */
                        idx = pf_purge_expired_states(idx, pf_hashmask /
                            (V_pf_default_rule.timeout[PFTM_INTERVAL] * 10));

                        /*
                         * Purge other expired types every
                         * PFTM_INTERVAL seconds.
                         */
                        if (idx == 0) {
                                /*
                                 * Order is important:
                                 * - states and src nodes reference rules
                                 * - states and rules reference kifs
                                 */
                                pf_purge_expired_fragments();
                                pf_purge_expired_src_nodes();
                                pf_purge_unlinked_rules();
                                pfi_kif_purge();
                        }
                        CURVNET_RESTORE();
                }
                VNET_LIST_RUNLOCK();
        }

        pf_end_threads++;
        sx_xunlock(&pf_end_lock);
        kproc_exit(0);
}

We iterate over all of our virtual pf instances (VNET_FOREACH()), check if it's active (for FreeBSD-EN-17.08, where we've seen this code before) and then check the expired states with pf_purge_expired_states(). We start at state 'idx' and only process a certain number (determined by the PFTM_INTERVAL setting) states. The pf_purge_expired_states() function returns a new idx value to tell us how far we got.

So, remember when I mentioned the odd V_ prefix? Those are per-vnet variables. They work a bit like thread-local variables. Each vnet (virtual network stack) keeps its state separate from the others, and the V_ variables use a pointer that's changed whenever we change the currently active vnet (say with CURVNET_SET() or CURVNET_RESTORE()). That's tracked in the 'curvnet' variable. In other words: there are as many V_pf_vnet_active variables as there are vnets: number of vnet jails plus one (for the host system).

Why is that relevant here? Note that idx is not a per-vnet variable, but we handle multiple pf instances here. We run through all of them in fact. That means that we end up checking the first X states in the first vnet, then check the second X states in the second vnet, the third X states in the third and so on and so on.

That of course means that we think we've run through all of the states in a vnet while we really only checked some of them. So when pf_purge_unlinked_rules() runs it can end up free()ing rules that actually are still in use because pf_purge_thread() skipped over the state(s) that actually used the rule.

The problem only happened if we reloaded rules in the host jail, because the active ruleset is never free()d, even if there are no states pointing to the rule.

That explains the panic, and the fix is actually quite straightforward: idx needs to be a per-vnet variable, V_pf_purge_idx, and then the problem is gone.

As is often the case, the solution to a fairly hard problem turns out to be really simple.

September 08, 2017

Here we go with a quick wrap-up of the second day. It started smoothly around 09:00 and was dedicated to more technical talks. After some refill of coffee, I was ready to follow all talks presented in the main track.

It started with LiveOverflow who presented “Play CTF“. CTF games (“Capture The Flag”) are present on the schedule of many infosec conferences but can also be independent events. The idea of the talk was original. It started with a short definition: They are security competitions with challenges that you must solve to find the “flag”. Sometimes, I’m playing CTF games but it’s always a dilemma. If you play, you don’t follow tracks (and I can’t write my wrap-ups!). Playing CTF is a great way to learn to hack. That’s what demonstrated Fabian in his talk. CTF’s are also a great to learn new technologies because it’s always changing. (example: many developers switched from PHP to Node.js). Challenges are usually based on typical vulnerabilities but you must be creative to solve them. They are often weird and do not always reflect the real world. So be prepared to fail :). The second part of the talk was more technical with examples of challenges. I like the one based on an issue present in Python 2 and how it compares objects. The second example was a format string vulnerability and finally a Python sandbox evasion. A very nice talk to start the day! If you’re interesting, you can find many CTF’s on a common agenda on ctftime.org.

The second slot was mine. I presented my overview of webshells from an HTTP protection perspective. Here are my slides:

Then, Ryan Lackey presented “The trouble with updates“. This is a fact, to be better protected against software vulnerabilities, patching is the key! Today, most operating systems and software have automatic update features but it’s not always for the good. Indeed, a few times a year, we read some bad news about a patch that broke a system or makes it less stable. But automatic installation also means that some bad code can be automatically injected into a system. What if the patching process is compromized? There was already several papers released about Microsoft WSUS! Some people also recommend to not install patches automatically. Certainly not on production systems. In this case, a best practice is to deploy the patches on a test system first to ensure that everything runs smoothly.

 

The next presentation was about “The status of web browsers VS DOM fuzzing” by Ivan Fratric (from the Google Project Zero). DOM or “Document Object Model” used in web browsers has been an interesting target for a while. A classic technique to find bugs in software is fuzzing. Ivan’s presentation reviewed how modern browsers are protecting themselves against fuzzing. Ivan explained how he developed his fuzzer and successfully used it to discover a lot of vulnerabilities. And guess what? All major browsers suffered from vulnerabilities.

I really expected a lot of the next talk about AutoIT by Vanja Svajcer from Cisco/Talos: “Hiding malware payloads with AutoIT”. A few days ago, I wrote a diary about AutoIT based malware. Vanja started with a nice introduction about AutoIT. This tool exists for years but seems to be back on stage. It’s a BASIC alike scripting language that can perform GUI automation and testing, can load external UDF (“User Defined Function, …). Of course, like any other languages, the code is never released as is, it is usually heavily obfuscated (variables and functions are renamed, junk code is inserted, strings split, etc…). It is even possible to inject the payload into an existing process. After the introduction, Vanja focused on a specific sample and explained how it infects the victim’s computer.

During the lunch break, I attended the lightning call session. A lot of interesting stuff and, amongst others, a quick presentation of Taler was performed by Sva. Taler is an alternative electronic payment system still under development. If you’re interested, check this website.

There was no talk foreseen in the afternoon, just the closing session and the results of the CTF. We spent the rest of the day chatting around some drinks under a nice weather. Yes, networking is also important during security conferences. This wraps up my first visit to FSEC. This is a nice event and the country looks nice. You can add put it on your wish-list of conferences to attend next year!

[The post FSEC 2017 Wrap-Up Day #2 has been first published on /dev/random]

As part of working in Acquia’s Office of the CTO, I’ve been working on the API-First Initiative for the past year and a half! Where are we at? Find out :)

Preview:

Intro


I have two Philips Hue lamps and mostly like them when  they are in the very warm white (2000k) color temperature setting.
The problem is that the setting does not persist when the lamps are switched using a physical power switch. To solve this I wrote the Hue Persistence program last year.
This program runs on the LAN and basically polls and stores the lamp settings and when it it detects a lamp going from off to on it restores the last setting.

After using it for a few weeks it became clear that there is one big problem with it: the Philips Hue hub sometimes accepts a connection from my program but then does not respond, yet keeps the connection open.
As the program is written using synchronous IO, this causes the program to hang forever, and it doesn't work anymore.

Solution


The Hue Persistence program uses the Rust Philips Hue library which using Hyper as a HTTP client to talk to the Philips Hue hub via it's restful API.
This library still uses Hyper 0.10 which does not provide async IO yet.

Recently there has been a lot of work for asynchronous IO in rust using Tokio and Futures. Version 0.11 of Hyper uses this.

First thing I had to do is port the Rust Philips Hue library to use Hyper 0.11 and thus async IO. This turned out to be quite an undertaking. The result of this can be found on the hyper_0.11 branch of my fork of the library. It is still a bit ugly, but works.

So far I've found using Futures for async IO to have the following issues:

  • error handling can be very confusing; error types need to be the same in future composition and you have to explicitly do error type conversions to make things work
  • future composition results in complex types, returning to Box<Future> helps a bit, but it still can be rather confusing (I didn't try the not yet released impl Trait solution)
  • it's not so obvious which future composition methods to use when, docs are sparse
  • branching (if, ...) inside futures is tedious again because of types having to match perfectly
In the end though I got the satisfaction of a working async library, after which I updated the Hue Persistence app to have a timeout on it's requests. This again turned out to be really tedious.
I had to use select2 with a sleep instead of tokio_timer::Timer::timeout() because it was just impossible to get the types to work for the error types when using tokio_timer::Timer, due to the TimeoutError containing Future type as a containing type in the signature of timeout().

Conclusion


My lamps are behaving fine now!

Async IO in Rust using Futures/tokio is interesting but at this point it still feels like "getting the types right" is somewhat in the way of normal coding productivity.
Some ergonomic improvements/language support could surely be used.
To be revisited in the Future (pun intended ;) ).



September 07, 2017

There are more and more infosec events worldwide and it’s always nice to attend new events and meet new people. This time, it is the case with FSEC. First visit to this security conference organized in Varazdin, Croatia. I had the honor to be invited as a speaker. This is already the seventh edition. FSEC was born thanks to the initiative of Tonimir Kisasondi. The event grew years after years and reached today 470 attendees (they reached the maximum capacity the venue). The conference was kicked off with some words by authorities and academic people. I was also impressed by the number of journalists presents to interview Tonimir! If only, we could have the same interest in Belgium! About the event itself, it is based on three tracks that cover large topics from round tables about GDPR to high technical presentations like browser exploration. Of course, there is also a CTF. The venue is a very nice  old theatre:

FSEC VenueCroatian National Theater, Varazdin

As usual, the event started with some keynotes. The first one was assigned to Alan Duric, the CEO of wire.com. The topic was “E2EE” or end-to-end encryption. Alan started with a review of the messaging tools that people used to communicate over the Internet (the “consumer world”). I liked the comparison of encryption with smoking in presence of children. Twenty years ago, parents were smoking in presence of their kids. Not because they were evil or bad parents, just because they were not aware of the risks associated to the smoke. It’s the same with encryption. In the past, people were not aware that their communications could be intercepted. Today communications are everywhere: not only people communicate but also computers (“M2M” or “Machine to machine”). Business critical data must also be protected, IoT, health data, etc… All of these must be federated with interoperability. Most of the data is stored in the cloud today. How to increase the awareness for E2EE requirements? Let’s imagine a deny of service attack against institutions from cars of video recorders. That’s not science-fiction! What if big cloud players are brought down due to DDoS? Our daily life will be affected. Security must be presented and sold in the right wat. It is driven by behavioral economics. Alan descrive three types of companies that are facing different challenges:

  • To protect intelectual property (risk is industrial espionage)
  • To protect client data (risks is coming with the GDPR)
  • Internal information (sensitive) like political parties (risk is cybercrime)

Again many services rely on service providers. Security and privacy are dependant on them and their practices and ability to fight threats. Interesting question from the audience: what to do when E2EE is not implemented? Do we wait or do we go with risks? Business can’t wait, the most important is to know the risks.

The second keynote speaker was Jaya Baloo, CISO of KPN. How KPN approach security? KPN is an old company (like most telco’s, they have to take into account multiple technologies, some of them being quite old). In 2012, they were hacked and realized that the question that you ask to yourself is not “if” but “when” you’ll be targeted. They provide critical services (emergency, airports, etc) Their mission: keep KPN secure, reliable and trusted. Besides a classic security life cycle (“prevent – detect – respond – verify”), they also have an internal red team responsible to test everything. As said Jaya: “They are there to hack all our sh*t!”, the message looks clear. If something is not secure, the project is simply blocked no matter  of what the business says. But the critical aspect for security professionals is to translate the impact to the business… What will happen if case of successful attack? How much will it cost to the organization? Jaya is known to be transparent and delivers good advice like “If your management relies on compliance, you’re in big sh*t!“. She also showed the security dashboard used by the management. Clear, simple, based on four categories: a “DEFCON” level, a threat intelligence, the vulnerabilities discovered by the red team and incidents. Very nice keynote full of good advices and facts!

After the lunch, I went to a session focussing on TLS: “TLS/SSL Transactions and PCI DSS, 3-D Secure, PSD2 & GDPR” presented by Predrag Kovačević. The idea was to review the status of the different TLS/SSL versions and then to see how to use the properly in different scenario’s: in a PCI/DSS environment, 3D-secure and GDPR. The problem was that the talk not in English… A good advice if you have to implement TLS: OWASP has a great cheat sheet about TLS. For the organizers, if a talk is not in English, it could be interesting to notify it on the schedule.

Then, I switched to another room to follow a presentation of the Sentinel SPS solution. This is a home router (for residential users) that has many interesting features all based on free software. It offers many interesting features: Internet access, multimedia features, IoT, telephone services and the “Sentinel services” that are:

  • A classic firewall
  • Network analysis
  • Hash lookup
  • RHA
  • YARA
  • Format parsing and analysis (check archives)
  • Parental controls
  • Remote management

Then, a talk looked interesting: “HiSiicon DVR hack” by Istvan Toth. Istvan made a very didactic presentation and explained how the firmware used in many DVR boxes was vulnerable. The presentation was easy to understand. It was like an hands-on presentation. He demonstrated how to use different tools to perform reverse engineering tasks (based on IDApro & python scripts or gdbserver). Step by step demonstration how to find the buffer overflow in the firmware. Nice presentation except that Istvan was reading every time his documentation 😉

Then, I followed Ivan Voras who presented “Designing a private blockchain… in case you really need one”. Everybody heard about block chains (mainly due to the BitCoin crypto currency) but not everybody knows how it works (and I’m part of this group). Only 10% of the audience looked to know what blockchain is (from a technical point of view). Ivan made a great introduction to the blockchain technology. Consider a block chain as some kind of database and data are dropped into blocks. In case of crypto currencies, one transactions is a block. Each new block signs the block before it. It’s immutable. But blockchains are not only used by crypto currencies, they’re many usages. Ivan developed his own private blockchain implementation that is a private blockchain. Each block is a SQLite database. The goal is to distribute official data in a safe way. His project is called Daisy and is available here.

Then, “Challenges and pitfalls of modern deep learning computing vision models” was presented by Marko Velic & Enes Deumic. Machine learning is a hot topic today and is used everywhere. We can find such technologies in security and surveillance, fraud detection, drones, self-driving cars, … After an introduction, they explained the risks that machine learning might face. Examples were based on self-driving cars. They demonstrated how the detection rate of road signs can be affected by covering some signs with stickers etc… Interesting. Again about machine learning, tgrzinic presented his machine learning framework to analyze malware samples. The framework is called MaliO and based on many free tools like Cuckoo.

The talk “Targeted attacks in 2017” by Felix Aimé.There were so many events that it’s not possible to remember all of them. Felix made a review of the last attacks and the techniques used to compromise the targets. They don’t change often and remains based on:

  • Compromised installation tools
  • Webmail phishing
  • Compromization of suppliers
  • Frontal compromise of servers
  • Classic spear phishing

sva came on stage to speak about PEP or “Pretty Easy Privacy”. I saw her presentation at the RMLL in July. The goal of PEP is got help the user to get rid of the complexity of PGP or S/MIME who are known to not be the best tools in terms of usability.

Finally, the last talk of the day (not the least one) was presented by Arnim Eijkhoudt from KPN about threat intelligence: “From incident to threat response“. I also gave the same message as Felix: “Sharing is caring”. Threat intelligence is:

  • Not a phase
  • Formalization with some caveats
  • Based on standards (STIX, TAXII, Cybox, )
  • Tons of players/feed/tools. All vendors have a feed
  • “Hype” (which could be bad)

The goal is to find the threat common to multiple incidents, to include a risk assessment component. It must be actionable and provides input for hunting. It must, of course, be a continuous process. Arnim explained different aspects of threat intelligence and finished with a real spear phishing case that targeted KPN (but without disclosing too much details ;-). A last advice: do not automatically classify  phishing emails as not malicious or to not submit they to VT. Perform a minimum set of controls to be certain that it’s not a targeted attack!

The day ended with a party but the music was way too loud, I had a dinner outside in a relaxed atmosphere with a friend. See you tomorrow for the next wrap-up!

[The post FSEC 2017 Wrap-Up Day #1 has been first published on /dev/random]

September 06, 2017

Just a quick blog post about an interesting sample that I found today. Usually, modern pieces of malware implement anti-debugging and anti-VM techniques. They perform some checks against the target and when a positive result is found, they silently exit… Such checks might be testing the screen resolution, the activity of a connected user, the presence of files on the desktop, etc. But they also search for interesting processes that could reveal that they are being monitored or debugged. This is achieved via the GetProcessesByName system call. Example:

processName = "tool_executed_by_analyst"
processList = Process.GetProcessesByName(processName)
If processList.Count > 0 Then
    ' Process is running, exit silently...
Else
    ' Process is not running, do our malicious stuff...
End If

This time, the sample did not search for running processes. Instead is a stealthy exit, it just executed a long list of taskkill.exe commands with process names like this:

taskkill.exe /IM <string> /T /F

“/IM” refers to the process image name, “/T” means to terminate all child processes and “/F” means to kill the process forcefully. This is a quite agressive technique!

Some processes are well-known, others were more exotic. Here is the full list:

avpmapp.exe
econceal.exe
escanmon.exe
escanpro.exe
TRAYSSER.EXE
TRAYICOS.EXE
econser.exe
VIEWTCP.EXE
FSHDLL64.exe
fsgk32.exe
fshoster32.exe
FSMA32.EXE
fsorsp.exe
fssm32.exe
FSM32.EXE
trigger.exe
FProtTray.exe
FPWin.exe
FPAVServer.exe
AVK.exe
GdBgInx64.exe
AVKProxy.exe
GDScan.exe
AVKWCtlx64.exe
AVKService.exe
AVKTray.exe
GDKBFltExe32.exe
GDSC.exe
virusutilities.exe
guardxservice.exe
guardxkickoff_x64.exe
iptray.exe
freshclam.exe
freshclamwrap.exe
K7RTScan.exe
K7FWSrvc.exe
K7PSSrvc.exe
K7EmlPxy.EXE
K7TSecurity.exe
K7AVScan.exe
K7CrvSvc.exe
K7SysMon.Exe
K7TSMain.exe
K7TSMngr.exe
nanosvc.exe
nanoav.exe
nnf.exe
nvcsvc.exe
nbrowser.exe
nseupdatesvc.exe
nfservice.exe
cmd.exetaskkill/IMnwscmon.exe
njeeves2.exe
nvcod.exe
nvoy.exe
zlhh.exe
Zlh.exe
nprosec.exe
Zanda.exe
NS.exe
acs.exe
op_mon.exe
PSANHost.exe
PSUAMain.exe
PSUAService.exe
AgentSvc.exe
BDSSVC.EXE
EMLPROXY.EXE
OPSSVC.EXE
ONLINENT.EXE
QUHLPSVC.EXE
SAPISSVC.EXE
SCANNER.EXE
SCANWSCS.EXE
scproxysrv.exe
ScSecSvc.exe
SUPERAntiSpyware.exe
SASCore64.exe
SSUpdate64.exe
SUPERDelete.exe
SASTask.exe
K7RTScan.exe
K7FWSrvc.exe
K7PSSrvc.exe
K7EmlPxy.EXE
K7TSecurity.exe
K7AVScan.exe
K7CrvSvc.exe
K7SysMon.Exe
K7TSMain.exe
K7TSMngr.exe
uiWinMgr.exe
uiWatchDog.exe
uiSeAgnt.exe
PtWatchDog.exe
PtSvcHost.exe
PtSessionAgent.exe
coreFrameworkHost.exe
coreServiceShell.exe
uiUpdateTray.exe
VIPREUI.exe
SBAMSvc.exe
SBAMTray.exe
SBPIMSvc.exe
bavhm.exe
BavSvc.exe
BavTray.exe
Bav.exe
BavWebClient.exe
BavUpdater.exe
MCShieldCCC.exe
MCShieldRTM.exe
MCShieldDS.exe
MCS-Uninstall.exe
SDScan.exe
SDFSSvc.exe
SDWelcome.exe
SDTray.exe
UnThreat.exe
utsvc.exe
FortiClient.exe
fcappdb.exe
FCDBlog.exe
FCHelper64.exe
fmon.exe
FortiESNAC.exe
FortiProxy.exe
FortiSSLVPNdaemon.exe
FortiTray.exe
FortiFW.exe
FortiClient_Diagnostic_Tool.exe
av_task.exe
CertReg.exe
FilMsg.exe
FilUp.exe
filwscc.exe
filwscc.exe
psview.exe
quamgr.exe
quamgr.exe
schmgr.exe
schmgr.exe
twsscan.exe
twssrv.exe
UserReg.exe

[The post Interesting List of Windows Processes Killed by Malicious Software has been first published on /dev/random]

Ever since I saw Magnolia I’ve been a fan of Aimee Mann. Not that I know all her songs, not that I listen to her music that regularly, but every time I do I regret having waited that long.

Anyway, she’s still going strong with high-quality songwriting about the not-so-pleasant aspects of life. This is her in the KEXP studios in July 2017, enjoy!

YouTube Video
Watch this video on YouTube.

DRY, standing for Don’t Repeat Yourself, is a well-known design principle in the software development world.

It is not uncommon for removal of duplication to take center stage via mantras such as “Repetition is the root of all evil”. Yet while duplication is often bad, the well intended pursuit of DRY often leads people astray. To see why, let’s take a step back and look at what we want to achieve by removing duplication.

The Goal of Software

First and foremost, software exists to fulfill a purpose. Your client, which can be your employer, is paying money because they want the software to provide value. As a developer it is your job to provide this value as effectively as possible. This includes tasks beyond writing code to do whatever your client specifies, and might best be done by not writing any code. The creation of code is expensive. Maintenance of code and extension of legacy code is even more so.

Since creation and maintenance of software is expensive, the quality of a developers work (when just looking at the code) can be measured in how quickly functionality is delivered in a satisfactory manner, and how easy to maintain and extend the system is afterwards. Many design discussions arise about trade-offs between those two measures. The DRY principle mainly situates itself in the latter category: reducing maintenance costs. Unfortunately applying DRY blindly often leads to increased maintenance costs.

The Good Side of DRY

So how does DRY help us reduce maintenance costs? If code is duplicated, and it needs to be changed, you will need to find all places where it is duplicated and apply the change. This is (obviously) more difficult than modifying one place, and more error prone. You can forget about one place where the change needs to be applied, you can accidentally apply it differently in one location, or you can modify code that happens to the same at present but should nevertheless not be changed due to conceptual differences (more on this later). This is also known as Shotgun Surgery. Duplicated code tends to also obscure the structure and intent of your code, making it harder to understand and modify. And finally, it conveys a sense of carelessness and lack of responsibility, which begets more carelessness.

Everyone that has been in the industry for a little while has come across horrid procedural code, or perhaps pretend-OO code, where copy-paste was apparently the favorite hammer of its creators. Such programmers indeed should heed DRY, cause what they are producing suffers from the issues we just went over. So where is The Fallacy of DRY?

The Fallacy of DRY

Since removal of duplication is a means towards more maintainable code, we should only remove duplication if that removal makes the code more maintainable.

If you are reading this, presumably you are not a copy-and-paste programmer. Almost no one I ever worked with is. Once you know how to create well designed OO applications (ie by knowing the SOLID principles), are writing tests, etc, the code you create will be very different from the work of a copy-paste-programmer. Even when adhering to the SOLID principles (to the extend that it makes sense) there might still be duplication that should be removed.The catch here is that this duplication will be mixed together with duplication that should stay, since removing it makes the code less maintainable. Hence trying to remove all duplication is likely to be counter productive.

Costs of Unification

How can removing duplication make code less maintainable? If the costs of unification outweigh the costs of duplication, then we should stick with duplication. We’ve already gone over some of the costs of duplication, such as the need for Shotgun Surgery. So let’s now have a look at the costs of unification.

The first cost is added complexity. If you have two classes with a little bit of common code, you can extract this common code into a service, or if you are a masochist extract it into a base class. In both cases you got rid of the duplication by introducing a new class. While doing this you might reduce the total complexity by not having the duplication, and such extracting might make sense in the first place for instance to avoid a Single Responsibility Principle violation. Still, if the only reason for the extraction is reducing duplication, ask yourself if you are reducing the overall complexity or adding to it.

Another cost is coupling. If you have two classes with some common code, they can be fully independent. If you extract the common code into a service, both classes will now depend upon this service. This means that if you make a change to the service, you will need to pay attention to both classes using the service, and make sure they do not break. This is especially a problem if the service ends up being extended to do more things, though that is more of a SOLID issue. I’ll skip going of the results of code reuse via inheritance to avoid suicidal (or homicidal) thoughts in myself and my readers.

DRY = Coupling

— A slide at DDDEU 2017

The coupling increases the need for communication. This is especially true in the large, when talking about unifying code between components or application, and when different teams end up depending on the same shared code. In such a situation it becomes very important that it is clear to everyone what exactly is expected from a piece of code, and making changes is often slow and costly due to the communication needed to make sure they work for everyone.

Another result of unification is that code can no longer evolve separately. If we have our two classes with some common code, and in the first a small behavior change is needed in this code, this change is easy to make. If you are dealing with a common service, you might do something such as adding a flag. That might even be the best thing to do, though it is likely to be harmful design wise. Either way, you start down the path of corrupting your service, which now turned into a frog in a pot of water that is being heated. If you unified your code, this is another point at which to ask yourself if that is still the best trade-off, or if some duplication might be easier to maintain.

You might be able to represent two different concepts with the same bit of code. This is problematic not only because different concepts need to be able to evolve individually, it’s also misleading to have only a single representation in the code, which effectively hides that you are dealing with two different concepts. This is another point that gains importance the bigger the scope of reuse. Domain Driven Design has a strategic pattern called Bounded Contexts, which is about the separation of code that represents different (sub)domains. Generally speaking it is good to avoid sharing code between Bounded Contexts. You can find a concrete example of using the same code for two different concepts in my blog post on Implementing the Clean Architecture, in the section “Lesson learned: bounded contexts”.

DRY is for one Bounded Context

— Eric Evans

Conclusion

Duplication itself does not matter. We care about code being easy (cheap) to modify without introducing regressions. Therefore we want simple code that is easy to understand. Pursuing removal of duplication as an end-goal rather than looking at the costs and benefits tends to result in a more complex codebase, with higher coupling, higher communication needs, inferior design and misleading code.

September 03, 2017

The post 2 interesting things happened in last cron.weekly’s newsletter appeared first on ma.ttias.be.

As usual, I sent out my weekly Linux & open source newsletter this Sunday. Lots of good content in there, you should probably have a read; cron.weekly issue #96: LogDevice, qmail, redis, Linus, HAProxy, libraries, concert, restic & more .

But as it was sent, I received a fair amount of feedback regarding 2 links I shared in the newsletter.

concert has been deprecated

I learned about concert, a tool for managing certificates via Let's Encrypt, via a colleague playing with Minio, the open source S3 compatible server.

I wrote the newsletter on Saturday, and by Sunday morning, this commit had landed;

+***DEPRECATED -- This project is deprecated and not maintained anymore.***
+***It is recommended all users use https://certbot.eff.org/ instead.***
+
Deprecate concert project (#27)

Interesting: between me discovering the project & adding it to the newsletter, it got deprecated.

Things move fast in open source.

`publicfile` HTTP server does not support "?querystring" arguments

Ok, quick summary: I linked to an article about qmail's security guarantee & bounty program, which hasn't been breached since as early as 1997. That's impressive.

So naturally, I wanted to include it in the newsletter. By default, I translate all links a bit, to get better tracking of the source/medium through Google Analytics. It still keeps the URLs readable -- as opposed to an actual click tracker, which would scramble all links. It transforms them like this;

Pretty harmless, usually. I added some logic that I don't try to add query strings to an URL if it already contains any, as not to accidentally break any functionality.

However, if you check the links above: the first one works, the second does not. I didn't see that one coming.

Turns out, that blog/page is hosted on a webserver called 'publicfile', created by D. J. Bernstein, who also created qmail & a lot of articles ultimately defining the internet as we know it today.

However, the webserver doesn't support query string arguments.

*Queue the security vs. flexibility/usability debate*

Lessons learned

A couple of gotcha's;

  1. Maybe visit each link again right before it goes out
  2. At least test the final new URLs it they still work

Those are things I did at the very beginning of the newsletter, but after a while it becomes routine and you start to take things for granted. That's when things get interesting!

The post 2 interesting things happened in last cron.weekly’s newsletter appeared first on ma.ttias.be.

September 02, 2017

Sucuri writes about their pov on performance:

We focus on useful metrics to optimize speed, like total time, not first byte or server response time.

Whereas Neil Patel writes:

When you’re in the weeds of optimizing your site speed, play smart. Instead of trying hard to push down your document complete, start rendering, or fully loaded page load time metrics (all of which are important), focus on the highest-impact metric first: TTFB.

Where TTFB = time to first byte. Needless to say I’m with Neil on this!

Allan twisted my arm in BSDNow episode 209 so I'll have to talk about FreeBSD-EN-17:08.pf.

First things first, so I have to point out that I think Allan misremembered things. The heroic debugging story is PR 219251, which I'll try to write about later.

FreeBSD-EN-17:08.pf is an issue that affected some FreeBSD 11.x systems, where FreeBSD would panic at startup. There were no reports for CURRENT. That looked like this:

Crash information

There's very little to go on here, but we do now the cause of the panic ("integer divide fault"), and that the current process was "pf purge". The pf purge thread is part of the pf housekeeping infrastructure. It's a housekeeping kernel thread which cleans up things like old states and expired fragments.

Here's the core loop:
void
pf_purge_thread(void *unused __unused)
{
        VNET_ITERATOR_DECL(vnet_iter);
        u_int idx = 0;

        sx_xlock(&pf_end_lock);
        while (pf_end_threads == 0) {
                sx_sleep(pf_purge_thread, &pf_end_lock, 0, "pftm", hz / 10);

                VNET_LIST_RLOCK();
                VNET_FOREACH(vnet_iter) {
                        CURVNET_SET(vnet_iter);

                        /*
                         *  Process 1/interval fraction of the state
                         * table every run.
                         */
                        idx = pf_purge_expired_states(idx, pf_hashmask /
                            (V_pf_default_rule.timeout[PFTM_INTERVAL] * 10));

                        /*
                         * Purge other expired types every
                         * PFTM_INTERVAL seconds.
                         */
                        if (idx == 0) {
                                /*
                                 * Order is important:
                                 * - states and src nodes reference rules
                                 * - states and rules reference kifs
                                 */
                                pf_purge_expired_fragments();
                                pf_purge_expired_src_nodes();
                                pf_purge_unlinked_rules();
                                pfi_kif_purge();
                        }
                        CURVNET_RESTORE();
                }
                VNET_LIST_RUNLOCK();
        }

        pf_end_threads++;
        sx_xunlock(&pf_end_lock);
        kproc_exit(0);
}

The lack of mention of pf functions in the backtrace is a hint unto itself. It suggests that the error is probably directly in pf_purge_thread(). It might also be in one of the static functions it calls, because compilers often just inline those so they don't generate stack frames.

Remember that the problem is an "integer divide fault". How can integer divisions be a problem? Well, you can try to divide by zero. The most obvious suspect for this is this code:

	idx = pf_purge_expired_states(idx, pf_hashmask /
		(V_pf_default_rule.timeout[PFTM_INTERVAL] * 10));

However, this variable is both correctly initialised (in pfattach_vnet()) and can only be modified through the DIOCSETTIMEOUT ioctl() call and that one checks for zero.

At that point I had no idea how this could happen, but because the problem did not affect CURRENT I looked at the commit history and found this commit from Luiz Otavio O Souza:

	Do not run the pf purge thread while the VNET variables are not
	initialized, this can cause a divide by zero (if the VNET initialization
	takes to long to complete).

	Obtained from:	pfSense
	Sponsored by:	Rubicon Communications, LLC (Netgate)

That sounds very familiar, and indeed, applying the patch fixed the problem. Luiz explained it well: it's possible to use V_pf_default_rule.timeout before it's initialised, which caused this panic.

To me, this reaffirms the importance of writing good commit messages: because Luiz mentioned both the pf purge thread and the division by zero I was easily able to find the relevant commit. If I hadn't found it this fix would have taken a lot longer.

As I've blogged before, I've been on and off working on SReview, a video review system. Development over the past year has been mostly driven by the need to have something up and running for first FOSDEM 2017, and then DebConf17, and so I've cut corners left and right which made the system, while functional, not quite entirely perfect everywhere. For instance, the backend scripts were done in ad-hoc perl, each reinventing their own wheel. Doing so made it easier for me to experiment with things and figure out where I want them to go, without immediately creating a lot of baggage that is not necessarily something I want to be stuck to. This flexibility has already paid off, in that I've redone the state machine between FOSDEM and DebConf17—and all it needed was to update a few SQL statements here and there. Well, and add a few of them, too.

It was always the intent to replace most of the ad-hoc perl with something better, however, once the time was ripe. One place where historical baggage is not so much of a problem, and where in fact abstracting away the complexity would now be an asset, is in the area of FFmpeg command lines. Currently, these are built by simple string expansion. For example, we do something like this (shortened for brevity):

system("ffmpeg -y -i $outputdir/$slug.ts -pass 1 -passlogfile ...");

inside an environment where the $outputdir and $slug variables are set in a perl environment. That works, but it has some downsides; e.g., adding or removing options based on which codecs we're using is not so easy. It would be much more flexible if the command lines were generated dynamically based on requested output bandwidth and codecs, rather than that they be hardcoded in the file. Case in point: currently there are multiple versions of some of the backend scripts, that only differ in details—mostly the chosen codec on the ffmpeg command line. Obviously this is suboptimal.

Instead, we want a way where video file formats can be autodetected, so that I can just say "create a file that uses encoder etc settings of this other file here". In addition, we also want a way where we can say "create a file that uses encoder etc settings of this other file here, except for these one or two options that I want to fine-tune manually". When I first thought about doing that about a year ago, that seemed complicated and not worth it—or at least not to that extent.

Enter Moose.

The Moose OO system for Perl 5 is an interesting way to do object orientation in Perl. I knew Perl supports OO, and I had heard about Moose, but never had looked into it, mostly because the standard perl OO features were "good enough". Until now.

Moose has a concept of adding 'attributes' to objects. Attributes can be set at object construction time, or can be accessed later on by way of getter/setter functions, or even simply functions named after the attribute itself (the default). For more complicated attributes, where the value may not be known until some time after the object has been created, Moose borrows the concept of "lazy" variables from Perl 6:

package Object;

use Moose;

has 'time' => (
    is => 'rw',
    builder => 'read_time',
    lazy => 1,
);

sub read_time {
    return localtime();
}

The above object has an attribute 'time', which will not have a value initially. However, upon first read, the 'localtime()' function will be called, the result is cached, and then (and on all further calls of the same function), the cached result will be returned. In addition, since the attribute is read/write, the time can also be written to. In that case, any cached value that may exist will be overwritten, and if no cached value exists yet, the read_time function will never be called. (it is also possible to clear values if needs be, so that the function would be called again).

We use this with the following pattern:

package SReview::Video;

use Moose;

has 'url' => (
    is => 'rw',
)

has 'video_codec' => (
    is => 'rw',
    builder => '_probe_videocodec',
    lazy => 1,
);

has 'videodata' => (
    is => 'bare',
    reader => '_get_videodata',
    builder => '_probe_videodata',
    lazy => 1,
);

has 'probedata' => (
    is => 'bare',
    reader => '_get_probedata',
    builder => '_probe',
    lazy => 1,
);

sub _probe_videocodec {
    my $self = shift;
    return $self->_get_videodata->{codec_name};
}

sub _probe_videodata {
    my $self = shift;
    if(!exists($self->_get_probedata->{streams})) {
        return {};
    }
    foreach my $stream(@{$self->_get_probedata->{streams}}) {
        if($stream->{codec_type} eq "video") {
            return $stream;
        }
    }
    return {};
}

sub _probe {
    my $self = shift;

    open JSON, "ffprobe -print_format json -show_format -show_streams " . $self->url . "|"
    my $json = "";
    while(<JSON>) {
        $json .= $_;
    }
    close JSON;
    return decode_json($json);
}

The videodata and probedata attributes are internal-use only attributes, and are therefore of the 'bare' type—that is, they cannot be read nor written to. However, we do add 'reader' functions that can be used from inside the object, so that the object itself can access them. These reader functions are generated, so they're not part of the object source. The probedata attribute's builder simply calls ffprobe with the right command-line arguments to retrieve data in JSON format, and then decodes that JSON file.

Since the passed JSON file contains an array with (at least) two streams—one for video and one for audio—and since the ordering of those streams depends on the file and is therefore not guaranteed, we have to loop over them. Since doing so in each and every attribute of the file we might be interested in would be tedious, we add a videodata attribute that just returns the data for the first found video stream (the actual source also contains a similar one for audio streams).

So, if you create an SReview::Video object and you pass it a filename in the url attribute, and then immediately run print $object->video_codec, then the object will

  • call ffprobe, and cache the (decoded) output for further use
  • from that, extract the video stream data, and cache that for further use
  • from that, extract the name of the used codec, cache it, and then return that name to the caller.

However, if the caller first calls $object->video_codec('h264'), then the ffprobe and most of the caching will be skipped, and instead the h265 data will be returned as video codec name.

Okay, so with a reasonably small amount of code, we now have a bunch of attributes that have defaults based on actual files but can be overwritten when necessary. Useful, right? Well, you might also want to care about the fact that sometimes you want to generate a video file that uses the same codec settings of this other file here. That's easy. First, we add another attribute:

has 'reference' => (
    is => 'ro',
    isa => 'SReview::Video',
    predicate => 'has_reference'
);

which we then use in the _probe method like so:

sub _probe {
    my $self = shift;

    if($self->has_reference) {
        return $self->reference->_get_probedata;
    }
    # original code remains here
}

With that, we can create an object like so:

my $video = SReview::Video->new(url => 'file.ts');
my $generated = SReview::Video->new(url => 'file2.ts', reference => $video);

now if we ask the $generated object what the value of its video_codec setting is without telling it ourselves first, it will use the $video object for its probed data, and use that.

That only misses generating the ffmpeg command line, but that's all fairly straightforward and therefore left as an exercise to the reader. Or you can cheat, and look it up.

We now invite proposals for main track presentations, developer rooms, stands and lightning talks. FOSDEM offers open source and free software developers a place to meet, share ideas and collaborate. Renowned for being highly developer-oriented, the event brings together some 8000+ geeks from all over the world. The eighteenth edition will take place on Saturday 3rd and Sunday 4th February 2018 at the usual location: ULB Campus Solbosch in Brussels. We will record and stream all main tracks, devrooms and lightning talks live. The recordings will be published under the same licence as all FOSDEM content (CC-BY). If, exceptionally,舰

I published the following diary on isc.sans.org: “AutoIT based malware back in the wild“.

One week ago I wrote a diary with an analysis of a malicious RAR archive that contained an AutoIT script. The technique was not new but I was curious to see if this was a one-shot or not. To search for juicy samples, VirusTotal Intelligence or “VTI” is a nice source. Thanks to the “Retro Hunt” feature, it is possible to search for specific samples that were submitted. The search conditions are based on YARA rules… [Read more]

[The post [SANS ISC] AutoIT based malware back in the wild has been first published on /dev/random]

August 31, 2017

Logo Proxmox VECe jeudi 21 septembre 2017 à 19h se déroulera la 61ème séance montoise des Jeudis du Libre de Belgique.

Le sujet de cette séance : Présentation de Proxmox VE

Thématique : sysadmin|virtualisation|stockage

Public : sysadmin|entreprises|étudiants

L’animateur conférencier : Alexandre Derumier (Odiso)

Lieu de cette séance : Université de Mons, Faculté Polytechnique, Site Houdain, Rue de Houdain, 9, auditoire 3 (cf. ce plan sur le site de l’UMONS, ou la carte OSM). Entrée par la porte principale, au fond de la cour d’honneur. Suivre le fléchage à partir de là.

La participation sera gratuite et ne nécessitera que votre inscription nominative, de préférence préalable, ou à l’entrée de la séance. Merci d’indiquer votre intention en vous inscrivant via la page http://jeudisdulibre.fikket.com/. La séance sera suivie d’un verre de l’amitié (le tout sera terminé au plus tard à 22h).

Les Jeudis du Libre à Mons bénéficient aussi du soutien de nos partenaires : CETIC, OpenSides, MeaWeb et Phonoid.

Si vous êtes intéressé(e) par ce cycle mensuel, n’hésitez pas à consulter l’agenda et à vous inscrire sur la liste de diffusion afin de recevoir systématiquement les annonces.

Pour rappel, les Jeudis du Libre se veulent des espaces d’échanges autour de thématiques des Logiciels Libres. Les rencontres montoises se déroulent chaque troisième jeudi du mois, et sont organisées dans des locaux et en collaboration avec des Hautes Écoles et Facultés Universitaires montoises impliquées dans les formations d’informaticiens (UMONS, HEH et Condorcet), et avec le concours de l’A.S.B.L. LoLiGrUB, active dans la promotion des logiciels libres.

Description : Proxmox VE est une distribution Linux qui permet, grâce à des outils libres comme Qemu-kvm et les conteneurs LXC, de mettre en place une plateforme de virtualisation bare-metal avec le clustering et la haute disponibilité. Proxmox VE offre diverses solutions de réseaux et stockage comme le LVM, NFS, iSCSI, Ceph, ZFS, …

La présentation abordera la creation et gestion de machine virtuelles et containers, la gestion du réseau et firewall, la migration à chaud des machines et stockage, le hotplug des périphériques (cpu,ram,disk) des machines, la gestion du stockage (snapshot, clones, backup),la gestion du cluster et de la Haute dispo. Je présenterais également ceph, une solution de stockage cluster, repliqué, qui peux être embarqué sur proxmox.

Une démonstration live sera faite sur les différents points abordés.

short bio : Ingénieur système et réseaux depuis près de 13 ans, Manager de l’equipe infrastructure chez Odiso, la cellule hosting de la société M6. je suis un passionné du logiciel Libre, contributeur sur proxmox depuis 2009 et formateur proxmox pour la France. Nous faisons tourner en production sur Proxmox, +- 3000 vms et containers.

Entertainment tonight

Entertainment Tonight is the number one entertainment news magazine in the world, and has been on the air for over 30 years. Fans around the world rely on Entertainment Tonight to receive news and updates on their favorite celebrities and stars. I recently discovered that the newest star featured on Entertainment Tonight was Drupal 8!

Entertainment Tonight's new Drupal 8 website, ETOnline.com, receives 19 million monthly unique visitors, making it the second most visited entertainment news website.

Chapter Three helped build the site. This project really helped Drupal 8 move forward because they ported many modules to Drupal 8 in the process. Check it out at http://www.etonline.com!

Entertainment tonight

Entertainment Tonight is the number one entertainment news magazine in the world, and has been on the air for over 30 years. Fans around the world rely on Entertainment Tonight to receive news and updates on their favorite celebrities and stars. I recently discovered that the newest star featured on Entertainment Tonight was Drupal 8!

Entertainment Tonight's new Drupal 8 website, ETOnline.com, receives 19 million monthly unique visitors, making it the second most visited entertainment news website.

Chapter Three helped build the site. This project really helped Drupal 8 move forward because they ported many modules to Drupal 8 in the process. Check it out at http://www.etonline.com!

August 30, 2017

Feedback we received after DebConf17 was that the quality of the released videos was rather low. Since I'd been responsible for encoding the videos, I investigated that, and found that I had used the video bitrate defaults of ffmpeg, which are rather low. So, I read some more documentation, and came up with better settings for the encoding to be done properly.

While at it, I found that the reason that VP9 encoding takes forever, as I'd found before, has everything to do with the same low-bitrate defaults, and is not an inherent problem to VP9 encoding. In light of that, I ran a test encode of a video with VP8 as well as VP9 encoding, and found that it shaves off some of the file size for about the same quality, with little to no increase in encoder time.

Unfortunately, the initial VP9 encoder settings that I used produced files that would play in some media players and in some browsers, but not in all of them. A quick email to the WebM-discuss mailinglist got me a reply, explaining that since our cameras used YUV422 encoding, which VP9 supports but some media players do not, and that since VP8 doesn't support that format, the result was VP9 files with YUV422 encoding, and VP8 ones with YUV420. The fix was to add a single extra command-line parameter to force the pixel format to YUV420 and tell ffmpeg to also downsample the video to that.

After adding another few parameters so as to ensure that some metadata would be added to the files as well, I've now told the system to re-encode the whole corpus of DebConf17 videos in both VP8 as well as VP9. It would appear that the VP9 files are, on average, about 1/4th to 1/3rd smaller than the equivalent VP8 files, with no apparent loss in quality.

Since some people might like them for whatever reason, I've not dropped the old lower-quality VP8 files, but instead moved them to an "lq" directory. They are usually about half the size of the VP9 files, but the quality is pretty terrible.

TLDR: if you downloaded videos from the debconf video archive, you may want to check whether it has been re-encoded yet (which would be the case if the state on that page is "done"), and if so, download them anew.

August 29, 2017

The post Release Notes Driven Development (RNDD) appeared first on ma.ttias.be.

I've been enjoying coding a lot again lately, working on small little side projects like DNS Spy or open source tools like my recent adventures in Go. But since it's just me and no one else, I have to prioritize certain features or tasks, which is where I found the concept of Release Notes Driven Development very interesting.

It's a topic that came up on the latest Under the Radar podcast too, which is what prompted this blogpost.

Prioritizing as a single developer or small team

My code is riddled with technical debt. From the very first PHP tools I wrote a decade ago to recent open source projects like my Varnish configurations, each and every one of them has known flaws and known improvements, up for grabs.

The thing is, I know those flaws. I know those technical debts. I'm aware of them, can workaround them and can fit them inside my head when developing new features so I know how not to break old behaviour of my code.

Most of my code also doesn't have unit tests for a similar reason. Not because I don't see the value in them, but because I can't prioritize them in good conscience.

Why? Because they make for lousy release notes.

The thing is: if I have a couple of hours to spare and work on my projects, I want something meaningful to come out of those hours. I want something to brag about on Twitter. To show to my users. To add a feature that's been in high demand.

Release notes that matter

Here's what most release notes end up reading;

  • Performance fixes and improvements
  • Minor bugfixes
  • Less crashes than before

I'm sure your users benefit from that, but here's a set of release notes that are worth a lot more.

  • Added support for feature X
  • Added new layout for Y
  • Improved readability and rendering on Z

The difference? Those are tangible, real, improvements for your users. Things users look forward to.

Sure, less crashes is good too. Sure, minor bugfixes are too. But that's what you expect from every release. Software should increase in quality.

What makes an actual difference? Adding features, extending functionality and solving user problems.

Write release notes first

What I try to do for my projects is come up with the release notes first. I might not call them release notes, per sé, but a list of "braggable features" to showcase. Every line of code I write from then on is to make that goal happen as efficiently as possible, without sacrificing existing functionality or stability.

The latest DNS Spy releases improved CAA scanning functionality, bugfixed several issues related to dynamic TTLs and fixed several rendering bugs where domains would show up as out-of-sync.

What I didn't do? Rewrite core parts of my app (even though in some parts, that would be needed, as I'm learning more about the Laravel framework and the shortcuts it offers), add unit tests or rewrite parts of the framework.

Should I? Yes, in due time, but they make for lousy release notes and right now, that's not a luxury I can afford.

The post Release Notes Driven Development (RNDD) appeared first on ma.ttias.be.

August 28, 2017

So you can remove the author-pages with an author.php file in your (child) theme, but what if you don’t want to touch the theme you ask? Well, I just added this code snippet to two of the sites I manage to stop user-enumeration (which can be done on any WordPress site by going to /index.php?author=1):

add_action('wp','no_author_page');
function no_author_page() {
  if (is_author()) {
    global $wp_query;
    $wp_query->set_404();
    status_header( 404 );
    get_template_part( 404 );
    exit();
  }
}

Disclaimer: the bulk of above code was shamelessly copy/ pasted from https://wordpress.stackexchange.com/a/27124

The post Restore one-click certificate viewing in Chrome appeared first on ma.ttias.be.

There's an additional Chrome setting you can toggle that allows you to view a website's certificate more easily, instead of going to the Inspector > Security > View Certificate panels.

By default, if you click on a certificate in the address bar, it looks like this;

 

There's no way to click through to the certificate. That "learn more" link takes you to a Google webpage explaining browser security, but nothing to see that certificate.

To enable that, do;

  1. Open a new tab and go to chrome://flags/#show-cert-link
  2. Enable that setting

Now, restart your Chrome.

Next time you click on the Certificate in the address bar, it looks like this.

The new "Certificate" link is added, which if you click it, opens the certificate dialog box immediately.

Much easier!

The post Restore one-click certificate viewing in Chrome appeared first on ma.ttias.be.

August 27, 2017

Ceci est le billet 45 sur 45 dans la série Printeurs

Il s’est écoulé une seconde et une éternité. Un silence infini s’est installé mais dans ma tête rugit la fureur et le bruit. Les émotions semblent poliment se céder le passage. Dois-je hurler de colère ? Trembler de peur ? Tomber à genoux de tristesse ?

Qu’ai-je vu ? Que s’est-il exactement passé ? Junior est-il vraiment mort ? Qui était Junior ? Que connaissais-je de lui ? Ai-je le droit à la tristesse ? Dois-je d’abord me préoccuper de sauvegarder ma propre vie ?

Peut-être qu’on s’habitue à la violence et la mort. Ou bien le corps est-il merveilleusement programmé pour se mettre en état de choc lorsque c’est nécessaire. J’ai du mal à déglutir mais c’est les yeux parfaitement secs que je me tourne vers Eva et Max. Aucun des deux n’a esquissé le moindre mouvement. Aucun ne semble exprimer la moindre émotion bien que, dans le cas de Max, le contraire aurait été particulièrement étonnant.

Aucun de nous n’a envie de prendre la parole. Nous respectons ce moment silencieux, en dehors du temps, cette unique et minimale cérémonie de dernier adieu. Intuitivement, nous savons que Junior cessera définitivement d’exister lorsque nous commencerons à continuer nos vies sans lui, lorsque nous accepterons son absence, sa métamorphose depuis un être vivant vers un simple souvenir. Et puis, inexorablement, une ultime transformation déjà commencée en oubli. De quelle couleur étaient ses yeux encore ? Avait-il un léger accent trainant ?

Le souvenir et le recueillement sont des conforts dont on ne reconnait la valeur que lorsqu’on en est privé.

Un claquement sec a retentit. La paroi dans mon dos s’est brusquement escamotée, révélant une formidable architecture de métal et de verre. Machinalement, nous suivons les balises lumineuses qui parcourent le plancher comme d’agiles vipères luminiscentes. Est-ce à dessein ? Les créatures de lumière nous emmènent sur une passerelle de verre suspendue par des câbles d’acier. Sous nos pieds plongent les entrailles du batiment, les poutres, les chemins, les câbles de toutes les épaisseurs, les myriades d’étincelles.

— On dirait un ordinateur, souligne Max de sa voix neutre pré-programmée.
— Chaque gadget, chaque accessoire est aujourd’hui un ordinateur, murmuré-je. Les bâtiments sont traditionnellement des ensembles de milliers d’ordinateurs. Mais des ordinateurs interconnectés ne forment-ils pas finalement un seul et unique ordinateur ?
— Un ordinateur capable de se débarrasser des corps étrangers. Un véritable être vivant, souligne Eva !

Un panneau lumineux semble clignoter devant nos yeux.

“Attention ! Vous accédez à une zone protégée. Vos implants et accessoires vont être rendus inopérants.”

Autour de la passerelle sur laquelle nous progressons, un tore métallique flotte silencieusement dans une danse aux apparences surnaturelles.

Machinalement, je tâte mes tempes, à la recherche de mes lunettes inexistantes. J’ai entendu parler de cette désactivation par choc électromagnétique. Cela ne m’inquiète pas, je n’ai plus rien d’électronique. Je veux faire un sourire à Eva mais son visage est déformé par la panique. Elle semble lutter contre un violent instinct de répulsion. Lorsque la voix de Max retentit.

— Merde, fait-il !

Je réalise seulement qu’il va être affecté.

— Max, fais demi-tour ! Attends-nous dehors !

Immobile, Max se tient debout. Avançant d’un pas, je lui tape sur l’épaule.

— Allez Max, ne…

Raide comme un piquet, le corps subtilement composé de chair et de métal s’écroule dans un fracas indescriptible.

— Max ?

Se mordant les poings, les yeux étrangement remplis de larmes de colère, Eva me regarde :

— Laisse tomber Nellio ! Tout… tout s’est arrêté. Son corps ne pouvait plus vivre sans assistance, il était plus robot qu’humain…
— Était ? Tu veux dire qu’il…
— Oui. Une décharge électromagnétique contrôlée du portique. Nous devons notre survie au simple fait d’être…

Elle tousse violemment.

— D’être complètement biologiques! hurle-t-elle.
– Mais… Qui peut bien prendre de telles mesures de sécurité ? Quel est l’intérêt d’une défense aussi impénétrable contre la vie biologique et électronique ?
— C’est peut-être la seule possibilité lorsque tu as des choses à cacher.
— C’est tout de même extrême, non ?
— Nellio, ouvre tes yeux biologiques ! Il n’existe plus un endroit sur terre où un drone microscopique ou un apprenti journaliste ne puisse s’insérer. Tes pensées les plus intimes sont connues par les publicitaires avant même que ton cerveau ne soit entré en action. Vous, les humains, êtes des machines prévisibles et déterministes. Une fois le mode de fonctionnement analysé et découvert, rien n’est plus facile que de faire faire à un humain une série d’actions aléatoires. En fait, il est plus facile de manipuler les humains que les atomes ! Vous êtes tellement simples !
— Nous ? Mais les humains sont tellement différents ! La variété, la richesse…
— Arrête, on dirait que tu récites un mantra. Pour un cerveau humain, les humains sont complexes, c’est vrai. Mais pour un ordinateur, il n’y a pas plus de différences entre deux humains qu’entre deux fourmis. Ils obéissent aux mêmes lois.

Je m’arrête un instant, le souffle coupé. Les images de l’agonie de Junior, de la mort subite de Max dansent devant mes yeux. Je me sens étrangement calme.

— Eva, s’il-te-plait, réponds à deux questions sans m’interrompre.

Elle me fixe d’un regard froid mais garde les lèvres serrées.

— Premièrement, en quoi ton histoire de fourmis explique-t-elle ces mesures de sécurité ?
— Ces mesures, comme tu dis, sont la seule solution pour permettre aux occupants de cet immeuble ne pas devenir une fourmi parmi les autres. Aucune information non-contrôlée ne peut sortir. Aucune influence ne peut pénétrer.
— Donc aucun être vivant, fut-il biologique, électronique ou un mélange des deux ne peut arriver jusqu’ici sans autorisation préalable. C’est d’une logique implacable. Et nous ne devons la vie qu’à une simple erreur de programmation, une faille dans le système de sécurité.

Les lèvres serrées, elle acquiesce tout en soutenant mon regard. Je ferme un instant les yeux, je réfléchis aux implications. Tout cela me dépasse, je suis un être terrorisé, en état de choc. Mon corps biologique est empli de molécules qui agissent en tout sens, activant différents signaux électriques que mon cerveau interprète machinalement : dors, protège-toi, fuis, découvre la vérité, cache-toi, sois-immobile, prépare-toi à combattre, réfléchis et comprends, pleure et appelle maman.

Mais ai-je encore seulement un choix à faire ? Mon destin n’est-il pas définitivement tracé ? Puis-je changer de direction ? Je me sens comme un automate, fatigué, épuisé, prêt à mourir pour retrouver le sommeil et l’apaisement.

Eva n’a pas bougé. Je lui murmure :
— Il nous reste à découvrir si cette faille était intentionnelle ou non…

 

Photo par DS.

Vous avez aimé votre lecture ? Soutenez l’auteur sur Tipeee, Patreon, Paypal ou Liberapay. Même un don symbolique fait toute la différence ! Retrouvons-nous ensuite sur Facebook, Medium, Twitter ou Mastodon.

Ce texte est publié sous la licence CC-By BE.

August 25, 2017

I published the following diary on isc.sans.org: “Malicious AutoIT script delivered in a self-extracting RAR file“.

Here is another sample that hit my curiosity. As usual, the infection vector was an email which delivered some HTML code in an attached file called “PO_5634_780.docx.html” (SHA1:d2158494e1b9e0bd85e56e431cbbbba465064f5a). It has a very low VT score (3/56) and contains a simple escaped Javascript code… [Read more]

[The post [SANS ISC] Malicious AutoIT script delivered in a self-extracting RAR file has been first published on /dev/random]

Mobile phones

I'm using my Oneplus One phone since 3 years, and my god, what a pleasant device this has been. And also a testimony to the rise and fall of Cyanmod Inc. It is currently the most used device with LineageOS, and will probably the first one to receive LOS15. People were laughing 3 years ago with OnePlus 'flagship killer' theme, but now, 3 years later, new mid- and highrange phones still carry 3 GB of RAM. So finding a worthy replacement was hard.

As I commute by train nowadays, I wanted to breach the 6 inch barrier to have a nice big screen, and battery life was extremely important. To make the jump from my Oneplus One reasonable, it should have at least 4GB of RAM. And only 2 devices seem to fit the above : the Xiaomi Mi Max 2 and the Mi Mix. I was lucky that the Mi Mix 2 was announced, as most webshops were dumping the Mi Mix at affordable prices. I quickly ordered one, and received it some weeks later.

This device has a surprisingly low footprint, as it's not much larger than a 5.7 inch phone, so it's quite portable. The ceramic glass is slippery, but not the bar of soap most of the web reviews make of it. The real problem is that the phone is slippery and heavy, so I'm a bit reluctant to use it without a case. Luckily, it comes with a nice leather premium case as well. The screen is fantastic, and it really is the star of this mobile phone. Viewing angles are great, vibrant colors, and nice outdoor readability.

So fantastic hardware, what about the software ? It came to my surprise with an unlocked bootloader, and with the MIUI Global ROM installed. TWRP installation was unsuccessful, as the touchscreen didn't worked, and I had to install a customized TWRP in order to get it working. Quite shocking, and being used to the openess of OPO, rather an unpleasant experience. I quickly installed the Xiaomi.EU ROM, as it sports full Dutch language support, and decided to try out MIUI during a week. MIUI is something people either love or hate, and after one week of usage, I must admit I fall into the latter group. I kept loosing myself into the settings screens, and could not live with the MIUI quirks. Especially notifications - you must enable notifications per application in 3 different settings screens - madness ! Notifications are broken in MIUI, but I'm convinced they are broken by design, to maximize battery life. And gosh, battery life is really impressive. I got 9 hours SOT with 3 days on a single charge with medium to heavy usage.

After a week, I wiped everything in favor of LineageOS. So, what do you loose when installing LOS onto the Xiaomi Mi Mix ? Battery life, camera functionalities and reduced fingerprint scanner accuracy. I guess I've lost 2 hours of SOT, while still having 3 to 4 days on a single battery charge with medium usage. Still impressive. Camera has lost 4K recording and slow-mo, though some camera mods re-enable them. LED seems brighter on LOS than in MIUI. Fingerprint scanner is slower and misses more scans - I regularly need 2 to 3 attempts to unlock the device.

So far, I'm really happy with the Mix. A tad too heavy, but really beautiful, and with lots of custom ROM support. After the Galaxy Nexus, the Oneplus One, the Xiaomi Mi Mix will be again a legendary phone I will enjoy using.

If you have children you should read this article, really!

A very small excerpt;

Even when a seismic event—a war, a technological leap, a free concert in the mud—plays an outsize role in shaping a group of young people, no single factor ever defines a generation. Parenting styles continue to change, as do school curricula and culture, and these things matter. But the twin rise of the smartphone and social media has caused an earthquake of a magnitude we’ve not seen in a very long time, if ever. There is compelling evidence that the devices we’ve placed in young people’s hands are having profound effects on their lives—and making them seriously unhappy.

So If you have children you should really read this article!

August 24, 2017

A few days ago I explained how we can do MVVM techniques like ICommand in Qt.

Today I’ll explain how to make and use a simple version of the, in the XAML MVVM world quite famous, RelayCommand. In the Microsoft Prism4 & 5 world this is DelegateCommand. Both are equivalent. I will only show a non-templated RelayCommand, so no RelayCommand<T> for now. Perhaps I’ll add a templated one to that mvvm project some other day.

What people call a delegate in C# is what C++ people call a Functor. Obviously we will use functors, then. Note that for people actually reading all those links: in C# the Action<T> and Func<T,G> are basically also C# delegates (or, functors, if you fancy C++’s names for this more).

Here is the RelayCommand.h:

#include <functional>
#include <QSharedPointer>
#include <MVVM/Commands/AbstractCommand.h>

class RelayCommand : public AbstractCommand
{
    Q_OBJECT
public:
    RelayCommand(std::function<void()> executeDelegatep,
                 std::function<bool()> canExecuteDelegatep,
                 QObject *parent = 0)
    : AbstractCommand(parent)
    , executeDelegate(executeDelegatep)
    , canExecuteDelegate(canExecuteDelegatep) {}

    void execute() Q_DECL_OVERRIDE;
    bool canExecute() const Q_DECL_OVERRIDE;
public slots:
    void evaluateCanExecute();
private:
    std::function<void()> executeDelegate;
    std::function<bool()> canExecuteDelegate;
};

The implementation is too simple to be true:

#include "RelayCommand.h"

bool RelayCommand::canExecute() const
{
    return canExecuteDelegate();
}

void RelayCommand::evaluateCanExecute()
{
    emit canExecuteChanged( canExecute() );
}

void RelayCommand::execute()
{
    executeDelegate();
}

Okay, so how do we use this? First we make a ViewModel. Because in this case we will define the command in C++. That probably means you want a ViewModel.

I added a CompositeCommand in the mix. For a Q_PROPERTY isn’t a CommandProxy really needed, as ownership stays in C++ (when for example you pass this as parent). For a Q_INVOKABLE you would need it to wrap the QSharedPointer<AbstractCommand>.

Note. I already hear you think: wait a minute, you are not passing this to the QObject’s constructor, it’s not a QScopedPointer and you have a new but no delete. That’s because CommandProxy converts the ownership rules to QQmlEngine::setObjectOwnership (this, QQmlEngine::JavaScriptOwnership) for itself. I don’t necessarily recommend its usage here (for it’s not immediately clear), but at the same time this is just a demo. You can try printing a warning in the destructor and you’ll see that the QML garbage collector takes care of it.

#include <QObject>
#include <QScopedPointer>

#include <MVVM/Commands/CommandProxy.h>
#include <MVVM/Commands/CompositeCommand.h>
#include <MVVM/Commands/RelayCommand.h>
#include <MVVM/Models/CommandListModel.h>

class ViewModel: public QObject
{
    Q_OBJECT

    Q_PROPERTY(CommandProxy* helloCommand READ helloCommand CONSTANT)
public:
    ViewModel(QObject *parent=0):QObject(parent),
        helloCmd(new CompositeCommand()){

        QSharedPointer<CompositeCommand> cCmd = helloCmd.dynamicCast<CompositeCommand>();
        cCmd->add( new RelayCommand ([=] { qWarning() << "Hello1 from C++ RelayCommand"; },
                            [=]{ return true; }));
        cCmd->add( new RelayCommand ([=] { qWarning() << "Hello2 from C++ RelayCommand"; },
                            [=]{ return true; }));
        proxyCmd = new CommandProxy (helloCmd);
    }
    CommandProxy* helloCommand() {
        return proxyCmd;
    }
private:
    QSharedPointer<AbstractCommand> helloCmd;
    CommandProxy *proxyCmd;
};

Let’s also make a very simple View.qml that uses the ViewModel

import QtQuick 2.3
import QtQuick.Window 2.0
import QtQuick.Controls 1.2

import Example 1.0

Item {
    property ViewModel viewModel: ViewModel {}

    Button {
        enabled: viewModel.helloCommand.canExecute
        onClicked: viewModel.helloCommand.execute()
    }
}

August 23, 2017

Yesterday I've switched to the gentoo-sources kernel package on Gentoo Linux. And with that, I also attempted (succesfully) to use the propriatary nvidia drivers so that I can enjoy both a smoother 3D experience while playing minecraft, as well as use the CUDA support so I don't need to use cloud-based services for small exercises.

The move to nvidia was quite simple, as the nvidia-drivers wiki article on the Gentoo wiki was quite easy to follow.

I published the following diary on isc.sans.org: “Malicious script dropping an executable signed by Avast?“.

Yesterday, I found an interesting sample that I started to analyze… It reached my spam trap attached to an email in Portuguese with the subject: “Venho por meio desta solicitar orçamento dos produtos” (“I hereby request the products budget”). There was one attached ZIP archive: PanilhaOrcamento.zip… [Read more]

[The post [SANS ISC] Malicious script dropping an executable signed by Avast? has been first published on /dev/random]

The post Removing root-owned files as a non-root user appeared first on ma.ttias.be.

Today I Learned: removing or renaming a file does not invoke the write() system call.

To be able to do anything with a file, the first step is to look it up in its directory. Listing a directory’s contents is controlled by the execute flag. If a user has execute permissions on a directory, he can see what’s inside it. Also, the execute flag on the directory gives access to its files’ inodes, which is crucial in this context, as the removal process unlinks the file.

Next, the removing part. Renaming or removing a file doesn’t involve the write() system call. Practically, we don’t need any permissions to remove the file, nor do we care about its owner. The only requirement is to have write permissions on the parent directory (and the execute flag on the parent directory).

Source: Casually removing root files

The post Removing root-owned files as a non-root user appeared first on ma.ttias.be.

August 22, 2017

Headless Drupal seems to be taking the world by storm. I'm currently in Sydney, and everyone I talked to so far, including the attendees at the Sydney Drupal User Group, is looking into headless Drupal. Digital agencies are experimenting with it on more projects, and there is even a new Decoupled Dev Days conference dedicated to the topic.

Roughly eight months ago, we asked ourselves in Acquia's Office of the CTO whether we could create a "headless" version of Drupal, optimized for integration with a variety of applications, channels and touchpoints. Such a version could help us build bridges with other developer communities working with different frameworks and programming languages, and the JavaScript community in particular.

I've been too busy with the transition at Acquia to blog about it in real time, but a few months ago, we released Reservoir. It's a Drupal-based content repository with all the necessary web service APIs needed to build decoupled front-end applications, be it a React application, an Ember front end, a native application, an augmented reality application, a Java or .NET application, or something completely different. You can even front-end it with a PHP application, something I hope to experiment with on my blog.

API-first distributions for Drupal like Reservoir and Contenta are a relatively new phenomenon but seem to be taking off rapidly. It's no surprise because an API-first approach is critical in a world where you have to operate agnostically across any channel and any form factor. I'm convinced that an API-first approach will be a critical addition to Drupal's future and could see a distribution like Reservoir or Contenta evolve to become a third installation profile for Drupal core (not formally decided).

Headless Drupal for both editors and developers

Reservoir welcome screenThe welcome screen after installing Reservoir.

The reason headless Drupal is taking off is that organizations are now grappling with a multitude of channels, including mobile applications, single-page JavaScript applications, IoT applications, digital signage, and content driven by augmented and virtual reality. Increasingly, organizations need a single place to house content.

What you want is an easy but powerful way for your editorial team to create and manage content, including administering advanced content models, content versioning, integrating media assets, translations, and more. All of that should be made easy through a great UI without having to involve a developer. This, incidentally, is aligned with Drupal 8's roadmap, in which we are focused on media management, workflows, layouts, and usability improvements through our outside-in work.

At the same time, you want to enable your developers to easily deliver that content to different devices, channels, and platforms. This means that the content needs to be available through APIs. This, too, is aligned with Drupal 8's roadmap, where we are focused on web services capabilities. Through Drupal's web service APIs, developers can build freely in different front-end technologies, such as Angular, React, Ember, and Swift, as well as Java and .NET. For developers, accomplishing this without the maintenance burden of a full Drupal site or the complexity of configuring standard Drupal to be decoupled is key.

API-first distributions like Reservoir keep Drupal's workflows and editorial UI intact but emphasize Drupal's web service APIs to return control to your developers. But with flexible content modeling and custom fields added to the equation, they also give more control over how editors can curate, combine, and remix content for different channels.

Success is getting to developer productivity faster

Reservoir side by side previews of HMTL and JSON APIReservoir includes side-by-side previews of content in HTML and JSON API output.

The goal of a content repository should be to make it simple for developers to consume your content, including digital assets and translations, through a set of web service APIs. Success means that a developer can programmatically access your content within minutes.

Reservoir tries to achieve this in four ways:

  1. Easy on-boarding. Reservoir provides a welcome tour with helpful guidance to create and edit content, map out new content models, manage access control, and most importantly, introspect the web service APIs you'll need to consume to serve your applications.
  2. JSON API standard. Reservoir makes use of JSON API, which is the specification used for many APIs in JSON and adopted by the Ember and Ruby on Rails communities. Using a common standard means you can on-board your developers faster.
  3. Great API documentation. Reservoir ships with great API documentation thanks to OpenAPI, formerly known as Swagger, which is a specification for describing an API. If you're not happy with the default documentation, you can bring your own approach by using Reservoir's OpenAPI export.
  4. Libraries, references, and SDKs. With the Waterwheel ecosystem, a series of libraries, references, and SDKs for popular languages like JavaScript and Swift, developers can skip learning the APIs and go straight to integrating Drupal content in their applications.

Next steps for Reservoir

Reservoir API documentation API documentation auto-generated based on the content model built in Reservoir.

We have a lot of great plans for Reservoir moving forward. Reservoir has several items on its short-term roadmap, including GraphQL support. As an emerging industry standard for data queries, GraphQL is a query language I first highlighted in my 2015 Barcelona keynote; see my blog post on the future of decoupled Drupal for a quick demo video.

We also plan to expand API coverage by adding the ability to programmatically manipulate users, tags, and other crucial content elements. This means that developers will be able to build richer integrations.

While content such as articles, pages, and other custom content types can be consumed and manipulated via web services today, upstream in Drupal core, API support for things like Drupal's blocks, menus, and layouts is in the works. The ability to influence more of Drupal's internals from external applications will open the door to better custom editorial interfaces.

Conclusion

I'm excited about Reservoir, not just because of the promise API-first distributions hold for the Drupal community, but because it helps us reach developers of different stripes who just need a simple content back end, all the while keeping all of the content editing functionality that editorial teams take for granted.

We've put the Reservoir codebase on GitHub, where you can open an issue, create a pull request, or contribute to documentation. Reservoir only advances when you give us feedback, so please let us know what you think!

Special thanks to Preston So for contributions to this blog post and to Ted Bowman, Wim Leers, and Matt Grill for feedback during the writing process.

Headless Drupal seems to be taking the world by storm. I'm currently in Sydney, and everyone I talked to so far, including the attendees at the Sydney Drupal User Group, is looking into headless Drupal. Digital agencies are experimenting with it on more projects, and there is even a new Decoupled Dev Days conference dedicated to the topic.

Roughly eight months ago, we asked ourselves in Acquia's Office of the CTO whether we could create a "headless" version of Drupal, optimized for integration with a variety of applications, channels and touchpoints. Such a version could help us build bridges with other developer communities working with different frameworks and programming languages, and the JavaScript community in particular.

I've been too busy with the transition at Acquia to blog about it in real time, but a few months ago, we released Reservoir. It's a Drupal-based content repository with all the necessary web service APIs needed to build decoupled front-end applications, be it a React application, an Ember front end, a native application, an augmented reality application, a Java or .NET application, or something completely different. You can even front-end it with a PHP application, something I hope to experiment with on my blog.

API-first distributions for Drupal like Reservoir and Contenta are a relatively new phenomenon but seem to be taking off rapidly. It's no surprise because an API-first approach is critical in a world where you have to operate agnostically across any channel and any form factor. I'm convinced that an API-first approach will be a critical addition to Drupal's future and could see a distribution like Reservoir or Contenta evolve to become a third installation profile for Drupal core (not formally decided).

Headless Drupal for both editors and developers

Reservoir welcome screenThe welcome screen after installing Reservoir.

The reason headless Drupal is taking off is that organizations are now grappling with a multitude of channels, including mobile applications, single-page JavaScript applications, IoT applications, digital signage, and content driven by augmented and virtual reality. Increasingly, organizations need a single place to house content.

What you want is an easy but powerful way for your editorial team to create and manage content, including administering advanced content models, content versioning, integrating media assets, translations, and more. All of that should be made easy through a great UI without having to involve a developer. This, incidentally, is aligned with Drupal 8's roadmap, in which we are focused on media management, workflows, layouts, and usability improvements through our outside-in work.

At the same time, you want to enable your developers to easily deliver that content to different devices, channels, and platforms. This means that the content needs to be available through APIs. This, too, is aligned with Drupal 8's roadmap, where we are focused on web services capabilities. Through Drupal's web service APIs, developers can build freely in different front-end technologies, such as Angular, React, Ember, and Swift, as well as Java and .NET. For developers, accomplishing this without the maintenance burden of a full Drupal site or the complexity of configuring standard Drupal to be decoupled is key.

API-first distributions like Reservoir keep Drupal's workflows and editorial UI intact but emphasize Drupal's web service APIs to return control to your developers. But with flexible content modeling and custom fields added to the equation, they also give more control over how editors can curate, combine, and remix content for different channels.

Success is getting to developer productivity faster

Reservoir side by side previews of HMTL and JSON APIReservoir includes side-by-side previews of content in HTML and JSON API output.

The goal of a content repository should be to make it simple for developers to consume your content, including digital assets and translations, through a set of web service APIs. Success means that a developer can programmatically access your content within minutes.

Reservoir tries to achieve this in four ways:

  1. Easy on-boarding. Reservoir provides a welcome tour with helpful guidance to create and edit content, map out new content models, manage access control, and most importantly, introspect the web service APIs you'll need to consume to serve your applications.
  2. JSON API standard. Reservoir makes use of JSON API, which is the specification used for many APIs in JSON and adopted by the Ember and Ruby on Rails communities. Using a common standard means you can on-board your developers faster.
  3. Great API documentation. Reservoir ships with great API documentation thanks to OpenAPI, formerly known as Swagger, which is a specification for describing an API. If you're not happy with the default documentation, you can bring your own approach by using Reservoir's OpenAPI export.
  4. Libraries, references, and SDKs. With the Waterwheel ecosystem, a series of libraries, references, and SDKs for popular languages like JavaScript and Swift, developers can skip learning the APIs and go straight to integrating Drupal content in their applications.

Next steps for Reservoir

Reservoir API documentation API documentation auto-generated based on the content model built in Reservoir.

We have a lot of great plans for Reservoir moving forward. Reservoir has several items on its short-term roadmap, including GraphQL support. As an emerging industry standard for data queries, GraphQL is a query language I first highlighted in my 2015 Barcelona keynote; see my blog post on the future of decoupled Drupal for a quick demo video.

We also plan to expand API coverage by adding the ability to programmatically manipulate users, tags, and other crucial content elements. This means that developers will be able to build richer integrations.

While content such as articles, pages, and other custom content types can be consumed and manipulated via web services today, upstream in Drupal core, API support for things like Drupal's blocks, menus, and layouts is in the works. The ability to influence more of Drupal's internals from external applications will open the door to better custom editorial interfaces.

Conclusion

I'm excited about Reservoir, not just because of the promise API-first distributions hold for the Drupal community, but because it helps us reach developers of different stripes who just need a simple content back end, all the while keeping all of the content editing functionality that editorial teams take for granted.

We've put the Reservoir codebase on GitHub, where you can open an issue, create a pull request, or contribute to documentation. Reservoir only advances when you give us feedback, so please let us know what you think!

Special thanks to Preston So for contributions to this blog post and to Ted Bowman, Wim Leers, and Matt Grill for feedback during the writing process.

You've might already read it on the Gentoo news site, the Hardened Linux kernel sources are removed from the tree due to the grsecurity change where the grsecurity Linux kernel patches are no longer provided for free. The decision was made due to supportability and maintainability reasons.

That doesn't mean that users who want to stick with the grsecurity related hardening features are left alone. Agostino Sarubbo has started providing sys-kernel/grsecurity-sources for the users who want to stick with it, as it is based on minipli's unofficial patchset. I seriously hope that the patchset will continue to be maintained and, who knows, even evolve further.

Personally though, I'm switching to the Gentoo sources, and stick with SELinux as one of the protection measures. And with that, I might even start using my NVidia graphics card a bit more, as that one hasn't been touched in several years (I have an Optimus-capable setup with both an Intel integrated graphics card and an NVidia one, but all attempts to use nouveau for the one game I like to play - minecraft - didn't work out that well).

I published the following diary on isc.sans.org: “Defang all the things!“.

Today, I would like to promote a best practice via a small Python module that is very helpful when you’re dealing with suspicious or malicious URLs. Links in documents are potentially dangerous because users can always click by mistake on them. Many automated tools and scripts are processing documents to fetch links. Even if the original document does not provide dynamic links, many applications will detect them and change them to real links… [Read more]

[The post [SANS ISC] Defang all the things! has been first published on /dev/random]

August 21, 2017

Just a quick post about an interesting file found in a phishing kit. Bad guys use common techniques to prevent crawlers, scanners or security companies from accessing their pages. Usually, they deploy a .htaccess file to achieve this. Today, I found a phishing kit related to a bank (ANZ) with such protection. But, in this case, the attackers took the time to comment out the blocked IP addresses and user-agents. Note that they also prevent other malicious traffic (like bots) to reach them. Very interesting! Want to know who’s blocked? Have a look at the file:

<Limit GET POST>
order allow,deny
deny from 209.85.32.23 # totaldomaindata (checkmark)
deny from 66.205.64.22
deny from 98.247.136.154
deny from 178.25.218.88
deny from 98.247.136.154
deny from 63.229.4.212
deny from 66.135.207.155
deny from 66.77.136.153
deny from 64.122.169.98
deny from 54.217.8.129
deny from 38.100.21.113
deny from 96.47.226.21
deny from 54.197.81.106
deny from 68.168.131.216
deny from 65.17.253.220
deny from 78.151.209.28
deny from 66.135.207.155
deny from 207.102.138.158
deny from 209.139.197.125
deny from 66.77.136.153
deny from 66.77.136.123
deny from 72.64.146.136
deny from 124.178.234.95
deny from 67.15.182.35
deny from 203.68. # taiwan academic network
deny from 218.58.124. # china jpg giftsite spammer
deny from 218.58.125.
deny from 62.194.7. # NE spambot
deny from 85.17.6. # netherlands
deny from 194.213. # czech norway sweden etc
deny from 64.27.2.18 # SEO masked as SE
deny from 64.27.2.19 # SEO masked as SE
deny from 212.187.116. # clown from Netherlands siphoning bible site
deny from 84.87. # clown from Netherlands siphoning bible site
deny from 222.252. # vietnam spammer
deny from 203.160.1. # vietnam spammer
deny from 82.60.1. # spamming Italy block
deny from 68.46.186.93 # clown on comcast
deny from 65.254.33.130 # unknown spain bot
deny from 82.131.195. # hungarian BS bot
deny from 217.153. # poland
deny from 202.108.252. # repeated merch spam!
deny from 82.208. # czech russia romania etc
deny from 193.47.80.41 # BW sucking bot
deny from 66.234.139.194 # bogus crawler
deny from 80.96. # romania
deny from 66.232.98.76 # unknown bot
deny from 38.112.6.182 # cosmixcorp.com
deny from 82.165.252.147 # unknown Java BW waster
deny from 67.79.102.28 # blacklisted spammer
deny from 220.181.26. # sohu bot
deny from 64.62.136.196 # unknown stealth bot
deny from 62.163. # netherlands
deny from 195.113. # czech
deny from 213.185.106. # nigeria
deny from 213.185.107. # nigeria
deny from 67.184.49.166 # blacklisted IP
deny from 219.95. # malaysia
deny from 66.221.106.76 # mydropbox.com
deny from 81.93.165. # norway bot
deny from 81.223.254. # austrian bs bot
deny from 87.123.74. # patwebbot
deny from 62.193.213. # french BS bot
deny from 86.120. # romania
deny from 86.121.
deny from 86.122.
deny from 86.123.
deny from 86.124.
deny from 86.125.
deny from 86.126.
deny from 86.127.
deny from 220.194.54. # BS bandwidth wasting bot
deny from 210.51.167. # BS bot
deny from 204.14.48. # stealth bots webhost etc
deny from 66.180.170.47 # development bot
deny from 217.160.75.202 # bot rips way too fast
deny from 84.12.54.237 # unknown clown UK
deny from 65.19.154.24 # stealth bandwidth hog
deny from 216.32.73.122 # stealth bot
deny from 63.160.77.236 # stealth bot
deny from 12.44.181.220 # unknown bot
deny from 12.44.172.92 # stealth bot
deny from 139.18.2. # findlinks bot
deny from 70.85.193.178 # unknown bot
deny from 82.80. # israel
deny from 82.81.
deny from 213.180.128. # poland
deny from 213.180.129.
deny from 213.180.130.
deny from 213.180.131.
deny from 66.150.55.230 # findwhat.com stealth bot
deny from 67.15.175.114 # unknown bot
deny from 217.113.244.119 # spanish SE
deny from 194.224.199. # private spanish server
deny from 81.19.66. # russia
deny from 213.176.126. # iran
deny from 208.223.208.181 # security-lab1.juniper.net
deny from 208.223.208.182
deny from 208.223.208.183
deny from 208.223.208.184
deny from 208.223.208.185
deny from 67.167.114.21 # BS law-x.com scraper site bot
deny from 194.44.42. # ukraine
deny from 209.203.192. # Expedite Marketing
deny from 209.203.193.
deny from 209.203.194.
deny from 209.203.195.
deny from 209.203.196.
deny from 209.203.197.
deny from 209.203.198.
deny from 209.203.199.
deny from 209.203.200.
deny from 209.203.201.
deny from 209.203.202.
deny from 209.203.203.
deny from 209.203.204.
deny from 209.203.205.
deny from 209.203.206.
deny from 209.203.207.
deny from 64.62.175. # unknown bandwidth sucker
deny from 219.136.171. # china unknown bot
deny from 216.150.24.122 # sonicleads.com spambot
deny from 216.150.24.123
deny from 210.14.32. # annoying philipines spammer
deny from 220.132.126. # taiwan useragent = 3
deny from 66.194.6. # websense.com bandwidth waster
deny from 12.17.130.27 # sitesucker
deny from 65.164.129.91
deny from 207.155.199.163
deny from 208.252.91.3
deny from 198.54. # south africa scams, spam, etc
deny from 66.132.132.63 # securityspace.com
deny from 81.18.32. # nigeria
deny from 81.18.33.
deny from 81.18.34.
deny from 81.18.35.
deny from 81.18.36.
deny from 81.18.37.
deny from 81.18.38.
deny from 81.18.39.
deny from 81.18.40.
deny from 81.18.41.
deny from 81.18.42.
deny from 81.18.43.
deny from 81.18.44.
deny from 81.18.45.
deny from 81.18.46.
deny from 81.18.47.
deny from 192.115.134. # Israel, hacker heaven
deny from 65.11.200.242 # direct revenue bot
deny from 65.75.128.30 # fotopages.com
deny from 204.8.168. # gator.com
deny from 204.8.169.
deny from 204.8.170.
deny from 204.8.171.
deny from 64.152.73.
deny from 66.111.48.80 # spambot from russia
deny from 68.211.2.61 # clown using site copier on books
deny from 64.42.84.70 # addresses.com spambot
deny from 67.127.13.70 # clown hitting with gethtmlcontents3 from secure site
deny from 80.230. # israel
deny from 80.250.32. # nigeria
deny from 80.250.33.
deny from 80.250.34.
deny from 80.250.35.
deny from 80.250.36.
deny from 80.250.37.
deny from 80.250.38.
deny from 80.250.39.
deny from 80.250.40.
deny from 80.250.41.
deny from 80.250.42.
deny from 80.250.43.
deny from 80.250.44.
deny from 80.250.45.
deny from 80.250.46.
deny from 80.250.47.
deny from 69.28.130. # quepasa.com
deny from 213.8. # israel
deny from 64.42.105. # unknown speed bot
deny from 141.85. # romania
deny from 128.238.55. # polybot
deny from 67.68.89. # unknown masking bot
deny from 66.36.242.25 # unknown bot
deny from 81.199. # israel nigeria etc
deny from 195.111. # hungary
deny from 192.115.106. # clown from Israel speed downloading
deny from 204.94.59. # brandimensions.com bandwidth waster
deny from 12.209.181.242 # speed ripping unknown agent
deny from 217.73. # romania ukraina russia etc
deny from 217.218. # iran
deny from 217.219. # iran
deny from 216.53.84.61 # mail.mccarter.com
deny from 169.132.149.100 # www.mccarter.com - new jersey law firm
deny from 213.226.16. # bulgaria
deny from 216.252.167. # idiot from Ghana demands free merch for many emails
deny from 65.102. # WebContent Internatioanl
deny from 216.163.255.1 # rpa.metlife.com bored employees
deny from 67.127.164.125 # DSL bandwidth waster
deny from 193.253.199. # france SE art-online.com bandwidth waster
deny from 80.179.254. # clown from Israel using downloader
deny from 64.37.103. # spambots and other non customers
deny from 69.61.12.100 # spambot from servershost.net
deny from 69.61.12.101
deny from 66.246.43.167
deny from 64.124.14. # markmonitor.com
deny from 38.144.36.11 # allresearch.com
deny from 38.144.36.12
deny from 38.144.36.13
deny from 38.144.36.14
deny from 38.144.36.15
deny from 38.144.36.16
deny from 38.144.36.17
deny from 206.28.72. # gettyimages.com bandwidth waster
deny from 206.28.73.
deny from 206.28.74.
deny from 206.28.75.
deny from 206.28.76.
deny from 206.28.77.
deny from 206.28.78.
deny from 206.28.79.
deny from 209.73.228.160 # allresearch.com
deny from 209.73.228.161
deny from 209.73.228.162
deny from 209.73.228.163
deny from 209.73.228.164
deny from 209.73.228.165
deny from 209.73.228.166
deny from 209.73.228.167
deny from 209.73.228.168
deny from 209.73.228.169
deny from 209.73.228.170
deny from 209.73.228.171
deny from 209.73.228.172
deny from 209.73.228.173
deny from 209.73.228.174
deny from 209.73.228.175
deny from 158.108. # thailand university
deny from 168.187. # kuwait ministry of communications
deny from 168.188. # korea university
deny from 66.207.120.221 # net-sweeper.com
deny from 66.207.120.222
deny from 66.207.120.223
deny from 66.207.120.224
deny from 66.207.120.225
deny from 66.207.120.226
deny from 66.207.120.227
deny from 66.207.120.228
deny from 66.207.120.229
deny from 66.207.120.230
deny from 66.207.120.231
deny from 66.207.120.232
deny from 66.207.120.233
deny from 66.207.120.234
deny from 66.207.120.235
deny from 167.24. # usaa.com and wastemylife.com p3p client
deny from 192.118.48.247 # icomverse.com (Israel, hacker heaven)
deny from 192.118.48.248
deny from 192.118.48.249
deny from 67.209.128. # clown from TX, wastes bandwidth, abusive feedback
deny from 12.148.209. # NameProtect.com bandwidth waster
deny from 12.148.196. # NameProtect.com bandwidth waster
deny from 212.19.205. # clown from Netherlands impersonating Webcrawler!
deny from 206.190.171.172 # markwatch.com bandwidth waster (4 IPs)
deny from 206.190.171.173
deny from 206.190.171.174
deny from 206.190.171.175
deny from 211.157.
deny from 211.74.
deny from 64.14.202.182
deny from 213.219.11.19
deny from 193.220.178. # abusive crawler from Benin
deny from 24.77.178.1 # abusive OK cable user
deny from 68.65.53.71 # unknown user (java1.4.0_03) slowly crawling whole site!
deny from 198.26.120.13 # unknown .MIL user (keeps hitting one page over and over!)
deny from 63.148.99. # Cyveillance.com bandwidth waster
deny from 65.118.41. # Cyveillance.com bandwidth waster
deny from 192.116.85. # abusive crawler, no ref, no ua, Israel?
deny from 62.119.21. # sweden including picsearch.com bot
deny from 80.179.100. # Israeli bot
deny from 80.248.64.50 # guestbook spambot
deny from 64.106.213. # some clown in Jersey, Russian name, hammering links page
deny from 62.220.103. # Iran
allow from all
</Limit>
RewriteEngine On
RewriteCond %{HTTP_USER_AGENT} ^-?$ [NC,OR] # blank user-agent
RewriteCond %{HTTP_USER_AGENT} "addresses\.com" [NC,OR] # spambot
RewriteCond %{HTTP_USER_AGENT} "agnitum" [NC,OR] # firewall sw from Cyprus
RewriteCond %{HTTP_USER_AGENT} aipbot [NC,OR]
RewriteCond %{HTTP_USER_AGENT} alkaline [NC,OR]
RewriteCond %{HTTP_USER_AGENT} "almaden" [NC,OR] # IBM unknown crawler
RewriteCond %{HTTP_USER_AGENT} amfibi [NC,OR] # spanish SE
RewriteCond %{HTTP_USER_AGENT} "anarchie" [NC,OR] # OD
RewriteCond %{HTTP_USER_AGENT} anonymous [NC,OR]
RewriteCond %{HTTP_USER_AGENT} "applewebkit" [NC,OR] # OD
RewriteCond %{HTTP_USER_AGENT} "art-online" [NC,OR] # France SE
RewriteCond %{HTTP_USER_AGENT} arikus [NC,OR] # voxel.net webhost
RewriteCond %{HTTP_USER_AGENT} "aspseek" [NC,OR] # unknown agent
RewriteCond %{HTTP_USER_AGENT} baidu [NC,OR] # chinese language SE
RewriteCond %{HTTP_USER_AGENT} "blackbox" [NC,OR] # HTML to JPG converter
RewriteCond %{HTTP_USER_AGENT} "bordermanager" [NC,OR] # Novell network controller iow workers goofing off
RewriteCond %{HTTP_USER_AGENT} botswana [NC,OR] # Unknown Agent
RewriteCond %{HTTP_USER_AGENT} "bravobrian" [NC,OR] # unknown agent
RewriteCond %{HTTP_USER_AGENT} bruinbot [NC,OR]
RewriteCond %{HTTP_USER_AGENT} btbot [NC,OR]
RewriteCond %{HTTP_USER_AGENT} "caddbot" [NC,OR] # classified ad bot
RewriteCond %{HTTP_USER_AGENT} ccubee [NC,OR] # czech crawler
RewriteCond %{HTTP_USER_AGENT} cfetch [NC,OR]
RewriteCond %{HTTP_USER_AGENT} cfnetwork [NC,OR]
RewriteCond %{HTTP_USER_AGENT} cherry.?picker [NC,OR] # spambot
RewriteCond %{HTTP_USER_AGENT} cjnetworkquality [NC,OR] # cj.com bot
RewriteCond %{HTTP_USER_AGENT} claria [NC,OR] # gator.com
RewriteCond %{HTTP_USER_AGENT} combine [NC,OR] # swedish harvester
RewriteCond %{HTTP_USER_AGENT} contactbot [NC,OR]
RewriteCond %{HTTP_USER_AGENT} convera [NC,OR] # convera.com
RewriteCond %{HTTP_USER_AGENT} ConveraCrawler [NC,OR] # convera.com
RewriteCond %{HTTP_USER_AGENT} cosmos [NC,OR] # xyleme.com bot
RewriteCond %{HTTP_USER_AGENT} cowbot [NC,OR] # korean naver bot
RewriteCond %{HTTP_USER_AGENT} cuill [NC,OR] # www.cuill.com
RewriteCond %{HTTP_USER_AGENT} crescent [NC,OR] # OD
RewriteCond %{HTTP_USER_AGENT} dattatec [NC,OR] # argentina bot
RewriteCond %{HTTP_USER_AGENT} deepak [NC,OR] # research bot from California
RewriteCond %{HTTP_USER_AGENT} dloader [NC,OR] # unknown downloader
RewriteCond %{HTTP_USER_AGENT} "^DA \d\.\d " [NC,OR] # OD
RewriteCond %{HTTP_USER_AGENT} "DTS Agent" [NC,OR] # OD
RewriteCond %{HTTP_USER_AGENT} "^download" [NC,OR] # OD
RewriteCond %{HTTP_USER_AGENT} diamond [NC,OR] # gator.com
RewriteCond %{HTTP_USER_AGENT} dtaagent [NC,OR] # bot grabs too fast
RewriteCond %{HTTP_USER_AGENT} dumbot [NC,OR]
RewriteCond %{HTTP_USER_AGENT} easydl [NC,OR] # OD
RewriteCond %{HTTP_USER_AGENT} e?mail.?(collector|magnet|reaper|siphon|sweeper|harvest|collect|wolf) [NC,OR] # spambots
RewriteCond %{HTTP_USER_AGENT} "Educate Search" [NC,OR] # guestbook spambot
RewriteCond %{HTTP_USER_AGENT} ejupiter [NC,OR] # pathetic SE
RewriteCond %{HTTP_USER_AGENT} entrieva [NC,OR]
RewriteCond %{HTTP_USER_AGENT} exava.com [NC,OR]
RewriteCond %{HTTP_USER_AGENT} experimental [NC,OR]
RewriteCond %{HTTP_USER_AGENT} expired [NC,OR]
RewriteCond %{HTTP_USER_AGENT} express [NC,OR] # OD
RewriteCond %{HTTP_USER_AGENT} extractor [NC,OR] # OD
RewriteCond %{HTTP_USER_AGENT} faxobot [NC,OR] # faxo.com
RewriteCond %{HTTP_USER_AGENT} "Fetch API Request" [NC,OR] # OD
RewriteCond %{HTTP_USER_AGENT} "fast firstpage retriever" [NC,OR] # OD
RewriteCond %{HTTP_USER_AGENT} "fetchbook\.info" [NC,OR]
RewriteCond %{HTTP_USER_AGENT} findexa [NC,OR] # norway SE
RewriteCond %{HTTP_USER_AGENT} findlinks [NC,OR] # german experimental bot
RewriteCond %{HTTP_USER_AGENT} findwhat [NC,OR]
RewriteCond %{HTTP_USER_AGENT} flashget [NC,OR] # OD
RewriteCond %{HTTP_USER_AGENT} FlickBot [NC,OR] # rude bot
RewriteCond %{HTTP_USER_AGENT} "Franklin Locator" [NC,OR] # guestbook spambot
RewriteCond %{HTTP_USER_AGENT} gais [NC,OR] # Chinese SE
RewriteCond %{HTTP_USER_AGENT} gazz/ [NC,OR] # Japanese language bot
RewriteCond %{HTTP_USER_AGENT} geobot [NC,OR] # spain bot
RewriteCond %{HTTP_USER_AGENT} gethtmlcontent [NC,OR] # OD
RewriteCond %{HTTP_USER_AGENT} getright [NC,OR] # OD
RewriteCond %{HTTP_USER_AGENT} girafabot [NC,OR] # girafa.com SE thingy
RewriteCond %{HTTP_USER_AGENT} giveramp [NC,OR]
RewriteCond %{HTTP_USER_AGENT} go.?zilla [NC,OR] # OD
RewriteCond %{HTTP_USER_AGENT} gonzo [NC,OR]
RewriteCond %{HTTP_USER_AGENT} grabber [NC,OR] # OD
RewriteCond %{HTTP_USER_AGENT} "green research" [NC,OR] # unknown bot
RewriteCond %{HTTP_USER_AGENT} "green research, inc." [NC,OR] # unknown bot
RewriteCond %{HTTP_USER_AGENT} gulper [NC,OR]
RewriteCond %{HTTP_USER_AGENT} harvest [NC,OR]
RewriteCond %{HTTP_USER_AGENT} hloader [NC,OR] # unknown downloader
RewriteCond %{HTTP_USER_AGENT} hoowwwer [NC,OR] # finnish SE
RewriteCond %{HTTP_USER_AGENT} html2jpg [NC,OR] # HTML to JPG converter
RewriteCond %{HTTP_USER_AGENT} htmlparser [NC,OR]
RewriteCond %{HTTP_USER_AGENT} "http generic" [NC,OR] # Unknown agent
RewriteCond %{HTTP_USER_AGENT} httpclient [NC,OR] # OD Webdown
RewriteCond %{HTTP_USER_AGENT} httprequest [NC,OR]
RewriteCond %{HTTP_USER_AGENT} httrack [NC,OR] # OD
RewriteCond %{HTTP_USER_AGENT} ia_archiver [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ichiro [NC,OR] # Japanese language bot (see gazz)
RewriteCond %{HTTP_USER_AGENT} "ie plagin" [NC,OR]
RewriteCond %{HTTP_USER_AGENT} "ie plugin" [NC,OR]
RewriteCond %{HTTP_USER_AGENT} imagefetch [NC,OR] # rude bot
RewriteCond %{HTTP_USER_AGENT} "Indy Library" [NC,OR] # spambot
RewriteCond %{HTTP_USER_AGENT} "Industry Program" [NC,OR] # guestbook spambot
RewriteCond %{HTTP_USER_AGENT} "^internet explorer$" [NC,OR] # BS agent
RewriteCond %{HTTP_USER_AGENT} ineturl [NC,OR]
RewriteCond %{HTTP_USER_AGENT} innerprise [NC,OR] # innerprise.net
RewriteCond %{HTTP_USER_AGENT} irlbot [NC,OR] # research bot
RewriteCond %{HTTP_USER_AGENT} ithenticate [NC,OR] # iThenticate spybot
RewriteCond %{HTTP_USER_AGENT} iupui [NC,OR] # Unknown research (spam?) bot
RewriteCond %{HTTP_USER_AGENT} java [NC,OR] # generic textbook bots
RewriteCond %{HTTP_USER_AGENT} jetbot [NC,OR] # Unknown private SE
RewriteCond %{HTTP_USER_AGENT} joedog [NC,OR]
RewriteCond %{HTTP_USER_AGENT} k2spider [NC,OR] # unknown bot
RewriteCond %{HTTP_USER_AGENT} kuloko [NC,OR] # kuloko.com
RewriteCond %{HTTP_USER_AGENT} lanshan [NC,OR]
RewriteCond %{HTTP_USER_AGENT} lcabotaccept [NC,OR] # unknown bot
RewriteCond %{HTTP_USER_AGENT} larbin [NC,OR] # unknown (spambot)
RewriteCond %{HTTP_USER_AGENT} lapozz [NC,OR] # BS hungarian bot
RewriteCond %{HTTP_USER_AGENT} law-x [NC,OR] # scraper site bot
RewriteCond %{HTTP_USER_AGENT} linksmanager [NC,OR] # linksmanager.com spambot
RewriteCond %{HTTP_USER_AGENT} linkwalker [NC,OR] # spambot
RewriteCond %{HTTP_USER_AGENT} lmcrawler [NC,OR]
RewriteCond %{HTTP_USER_AGENT} lmqueuebot [NC,OR]
RewriteCond %{HTTP_USER_AGENT} loopimprovements [NC,OR]
RewriteCond %{HTTP_USER_AGENT} "lwp\:\:simple" [NC,OR]
RewriteCond %{HTTP_USER_AGENT} "lwp-trivial" [NC,OR]
RewriteCond %{HTTP_USER_AGENT} "Mac Finder" [NC,OR] # guestbook spambot
RewriteCond %{HTTP_USER_AGENT} "Microsoft URL Control" [NC,OR] # spambot
RewriteCond %{HTTP_USER_AGENT} "mister pix" [NC,OR] # rude bot
RewriteCond %{HTTP_USER_AGENT} "missauga" [NC,OR] # guestbook spambot
RewriteCond %{HTTP_USER_AGENT} "missigua" [NC,OR] # guestbook spambot
RewriteCond %{HTTP_USER_AGENT} madlyrics [NC,OR] # Winamp downloader
RewriteCond %{HTTP_USER_AGENT} marvin [NC,OR] # danish/whoever bot
RewriteCond %{HTTP_USER_AGENT} microsoftprototypecrawler [NC,OR]
RewriteCond %{HTTP_USER_AGENT} minirank [NC,OR]
RewriteCond %{HTTP_USER_AGENT} miva [NC,OR]
RewriteCond %{HTTP_USER_AGENT} mizzu [NC,OR] # Mizzu Labs bot
RewriteCond %{HTTP_USER_AGENT} mj12 [NC,OR]
RewriteCond %{HTTP_USER_AGENT} majestic [NC,OR]
RewriteCond %{HTTP_USER_AGENT} mogren [NC,OR] # russian bot
RewriteCond %{HTTP_USER_AGENT} "mozilla\(ie compatible\)" [NC,OR] # BS agent
RewriteCond %{HTTP_USER_AGENT} MSIECrawler [NC,OR] # IE's "make available offline" mode
RewriteCond %{HTTP_USER_AGENT} MSFrontPage [NC,OR] # OD
RewriteCond %{HTTP_USER_AGENT} msrbot [NC,OR]
RewriteCond %{HTTP_USER_AGENT} msproxy [NC,OR] # discontinued proxy software
RewriteCond %{HTTP_USER_AGENT} msx [NC,OR] # unknown agent
RewriteCond %{HTTP_USER_AGENT} mvaclient [NC,OR]
RewriteCond %{HTTP_USER_AGENT} "my session" [NC,OR] # unknown agent
RewriteCond %{HTTP_USER_AGENT} "NASA Search" [NC,OR] # bogus clown on comcast
RewriteCond %{HTTP_USER_AGENT} netresearchserver [NC,OR]
RewriteCond %{HTTP_USER_AGENT} netsprint [NC,OR]
RewriteCond %{HTTP_USER_AGENT} netwhat [NC,OR]
RewriteCond %{HTTP_USER_AGENT} nextgensearch [NC,OR] # BW waster
RewriteCond %{HTTP_USER_AGENT} nusearch [NC,OR] # spider OD
RewriteCond %{HTTP_USER_AGENT} nutch [NC,OR] # experimental bot
RewriteCond %{HTTP_USER_AGENT} ocelli [NC,OR] # www.globalspec.com
RewriteCond %{HTTP_USER_AGENT} offline [NC,OR] # OD
RewriteCond %{HTTP_USER_AGENT} omniexplorer [NC,OR] # useless bot
RewriteCond %{HTTP_USER_AGENT} "onsinn.de" [NC,OR]
RewriteCond %{HTTP_USER_AGENT} outfoxbot [NC,OR]
RewriteCond %{HTTP_USER_AGENT} nameprotect [NC,OR] # NameProtect spybot
RewriteCond %{HTTP_USER_AGENT} naver [NC,OR] # Korean robot
RewriteCond %{HTTP_USER_AGENT} net.?(ants|mechanic|spider|vampire|zip) [NC,OR] # ODs
RewriteCond %{HTTP_USER_AGENT} netcaptor [NC,OR] # OD
RewriteCond %{HTTP_USER_AGENT} nicebot [NC,OR] # stealth bot
RewriteCond %{HTTP_USER_AGENT} nicerspro [NC,OR] # spambot
RewriteCond %{HTTP_USER_AGENT} ninja [NC,OR] # Download Ninja OD
RewriteCond %{HTTP_USER_AGENT} nobody [NC,OR] # Unknown Agent
RewriteCond %{HTTP_USER_AGENT} noxtrum [NC,OR] # spanish private server
RewriteCond %{HTTP_USER_AGENT} NPBot [NC,OR] # NameProtect spybot
RewriteCond %{HTTP_USER_AGENT} "\ obot" [NC,OR] # Unknown bot
RewriteCond %{HTTP_USER_AGENT} "^obot$" [NC,OR] # Unknown bot
RewriteCond %{HTTP_USER_AGENT} openfind [NC,OR] # taiwan bot
RewriteCond %{HTTP_USER_AGENT} panopy [NC,OR] # unknown bot
RewriteCond %{HTTP_USER_AGENT} patwebbot [NC,OR] # bs bot from germany
RewriteCond %{HTTP_USER_AGENT} peerfactor [NC,OR]
RewriteCond %{HTTP_USER_AGENT} pipeline [NC,OR] # cable account based SE
RewriteCond %{HTTP_USER_AGENT} plink [NC,OR] # stealth bot
RewriteCond %{HTTP_USER_AGENT} "program shareware" [NC,OR] # guestbook spambot
RewriteCond %{HTTP_USER_AGENT} plantynet [NC,OR] # Korean bot
RewriteCond %{HTTP_USER_AGENT} "poe-component-client" [NC,OR]
RewriteCond %{HTTP_USER_AGENT} "polybot" [NC,OR] # cis.poly.edu
RewriteCond %{HTTP_USER_AGENT} psbot [NC,OR] # Picture Downloader
RewriteCond %{HTTP_USER_AGENT} picsearch [NC,OR] # Picture Downloader
RewriteCond %{HTTP_USER_AGENT} qarp [NC,OR]
RewriteCond %{HTTP_USER_AGENT} qcreep [NC,OR] # quepasa in disguise
RewriteCond %{HTTP_USER_AGENT} quepasa [NC,OR] # SouthAmerican bot
RewriteCond %{HTTP_USER_AGENT} "safari" [NC,OR] # OD
RewriteCond %{HTTP_USER_AGENT} "^sew$" [NC,OR] # unknown agent
RewriteCond %{HTTP_USER_AGENT} rampybot [NC,OR]
RewriteCond %{HTTP_USER_AGENT} research [NC,OR]
RewriteCond %{HTTP_USER_AGENT} sbider [NC,OR]
RewriteCond %{HTTP_USER_AGENT} schibstedsok [NC,OR]
RewriteCond %{HTTP_USER_AGENT} "scientec.de" [NC,OR]
RewriteCond %{HTTP_USER_AGENT} scspider [NC,OR] # SpamBot
RewriteCond %{HTTP_USER_AGENT} scumbot [NC,OR]
RewriteCond %{HTTP_USER_AGENT} search-o-rama [NC,OR]
RewriteCond %{HTTP_USER_AGENT} searchsight [NC,OR]
RewriteCond %{HTTP_USER_AGENT} searchwarp [NC,OR]
RewriteCond %{HTTP_USER_AGENT} seekbot [NC,OR]
RewriteCond %{HTTP_USER_AGENT} seznambot [NC,OR] # czech bot
RewriteCond %{HTTP_USER_AGENT} shim-crawler [NC,OR]
RewriteCond %{HTTP_USER_AGENT} siphon [NC,OR]
RewriteCond %{HTTP_USER_AGENT} sitemapper [NC,OR]
RewriteCond %{HTTP_USER_AGENT} sitesell [NC,OR]
RewriteCond %{HTTP_USER_AGENT} skywalker [NC,OR]
RewriteCond %{HTTP_USER_AGENT} sleuth [NC,OR]
RewriteCond %{HTTP_USER_AGENT} SlySearch [NC,OR] # SlySearch spybot
RewriteCond %{HTTP_USER_AGENT} snagger [NC,OR] # OD
RewriteCond %{HTTP_USER_AGENT} societyrobot [NC,OR]
RewriteCond %{HTTP_USER_AGENT} "sohu agent" [NC,OR] # spambot
RewriteCond %{HTTP_USER_AGENT} sohu-search [NC,OR] # spambot
RewriteCond %{HTTP_USER_AGENT} sonicquest [NC,OR]
RewriteCond %{HTTP_USER_AGENT} spider_pro [NC,OR] # innerprise.net
RewriteCond %{HTTP_USER_AGENT} spiderku [NC,OR]
RewriteCond %{HTTP_USER_AGENT} spiderman [NC,OR]
RewriteCond %{HTTP_USER_AGENT} sproose [NC,OR]
RewriteCond %{HTTP_USER_AGENT} sqworm [NC,OR] # unknown bot
RewriteCond %{HTTP_USER_AGENT} stackrambler [NC,OR] # russian bot
RewriteCond %{HTTP_USER_AGENT} steeler [NC,OR] # OD
RewriteCond %{HTTP_USER_AGENT} SurveyBot [NC,OR] # rude bot
RewriteCond %{HTTP_USER_AGENT} szukacz [NC,OR] # OD
RewriteCond %{HTTP_USER_AGENT} tcf [NC,OR]
RewriteCond %{HTTP_USER_AGENT} tele(port|soft) [NC,OR] # OD
RewriteCond %{HTTP_USER_AGENT} "test/0" [NC,OR]
RewriteCond %{HTTP_USER_AGENT} "test1" [NC,OR]
RewriteCond %{HTTP_USER_AGENT} "test 1" [NC,OR]
RewriteCond %{HTTP_USER_AGENT} "test rig" [NC,OR]
RewriteCond %{HTTP_USER_AGENT} "tsw bot" [NC,OR]
RewriteCond %{HTTP_USER_AGENT} terrawiz [NC,OR] # India SE
RewriteCond %{HTTP_USER_AGENT} trademark [NC,OR] # bandwidth waster trademarktracker.com
RewriteCond %{HTTP_USER_AGENT} transgenikbot [NC,OR]
RewriteCond %{HTTP_USER_AGENT} Turnitin [NC,OR] # Turnitin spybot
RewriteCond %{HTTP_USER_AGENT} twiceler [NC,OR] # www.cuill.com
RewriteCond %{HTTP_USER_AGENT} twotrees [NC,OR] # willow internet crawler
RewriteCond %{HTTP_USER_AGENT} "under the rainbow" [NC,OR] # unknown bot
RewriteCond %{HTTP_USER_AGENT} "unknown origin" [NC,OR] # unknown bot
RewriteCond %{HTTP_USER_AGENT} unchaos [NC,OR] # SE that spams web logs
RewriteCond %{HTTP_USER_AGENT} url2file [NC,OR] # OD
RewriteCond %{HTTP_USER_AGENT} usyd-nlp [NC,OR] # research spider
RewriteCond %{HTTP_USER_AGENT} "vb openurl" [NC,OR]
RewriteCond %{HTTP_USER_AGENT} visvo [NC,OR]
RewriteCond %{HTTP_USER_AGENT} votay [NC,OR]
RewriteCond %{HTTP_USER_AGENT} voyager [NC,OR]
RewriteCond %{HTTP_USER_AGENT} w3crobot [NC,OR]
RewriteCond %{HTTP_USER_AGENT} w3mir [NC,OR] # site copier
RewriteCond %{HTTP_USER_AGENT} wbdbot [NC,OR] #sky.siraza.net
RewriteCond %{HTTP_USER_AGENT} weasel [NC,OR]
RewriteCond %{HTTP_USER_AGENT} weazel [NC,OR]
RewriteCond %{HTTP_USER_AGENT} web.?(auto|bandit|collector|copier|devil|downloader|fetch|hook|mole|miner|mirror|reaper|sauger|sucker|site|snake|stripper|weasel|zip) [NC,OR] # ODs
RewriteCond %{HTTP_USER_AGENT} webclipping [NC,OR] # bandwidth waster webclipping.com
RewriteCond %{HTTP_USER_AGENT} webbug [NC,OR]
RewriteCond %{HTTP_USER_AGENT} webcollage [NC,OR]
RewriteCond %{HTTP_USER_AGENT} webindexer [NC,OR] # development bot
RewriteCond %{HTTP_USER_AGENT} webpix [NC,OR]
RewriteCond %{HTTP_USER_AGENT} webrace [NC,OR] # crawler
RewriteCond %{HTTP_USER_AGENT} webspider [NC,OR]
RewriteCond %{HTTP_USER_AGENT} websquash [NC,OR] # SEO
RewriteCond %{HTTP_USER_AGENT} "wells search" [NC,OR] # spambot
RewriteCond %{HTTP_USER_AGENT} "wep search" [NC,OR] # spambot
RewriteCond %{HTTP_USER_AGENT} wget [NC,OR] # OD
RewriteCond %{HTTP_USER_AGENT} wise-guys.nl [NC,OR] # Clown in NL
RewriteCond %{HTTP_USER_AGENT} "www.abot.com" [NC,OR]
RewriteCond %{HTTP_USER_AGENT} xirq [NC,OR]
RewriteCond %{HTTP_USER_AGENT} yottashopping [NC,OR]
RewriteCond %{HTTP_USER_AGENT} zao/ [NC,OR] # experimental Japan crawler
RewriteCond %{HTTP_USER_AGENT} zedzo [NC,OR]
RewriteCond %{HTTP_USER_AGENT} zeus [NC,OR]
RewriteCond %{HTTP_USER_AGENT} zspider [NC,OR]
RewriteCond %{HTTP_REFERER} iaea.org [NC,OR] # spam bot
RewriteCond %{HTTP_REFERER} wizard.yellowbrick.oz [NC,OR] # spam bot
RewriteCond %{HTTP_REFERER} brandimensions [NC,OR] # bandidth waster
RewriteCond %{HTTP_REFERER} imgurl= [NC,OR]
RewriteCond %{HTTP_REFERER} imgrefurl= [NC,OR]
RewriteCond %{REMOTE_ADDR} ^193.95.([1-2][0-9][0-9]). [NC,OR] # slovenia etc
RewriteCond %{REMOTE_ADDR} ^203.147.([0-4][0-9]). [NC,OR] # thailand
RewriteCond %{REMOTE_ADDR} ^80.87.([3-9][0-9]). [NC,OR] # ghana russia etc
RewriteCond %{REMOTE_ADDR} ^80.88.(1[0-5][0-9]). [NC,OR]
RewriteCond %{REMOTE_ADDR} ^203.87.(1[2-9][0-9]). [NC,OR] # philippines
RewriteCond %{REMOTE_ADDR} ^218.(1[0-9][0-9]). [NC,OR] # china korea
RewriteCond %{REMOTE_ADDR} ^211.([1-9][0-9]). [NC,OR] # china korea
RewriteCond %{REMOTE_ADDR} ^66.150.55.(2[2-3][0-9]). [NC,OR] # findwhat.com stealth bot
RewriteCond %{REMOTE_ADDR} ^64.110.([4-9][0-9]). [NC,OR]
RewriteCond %{REMOTE_ADDR} ^64.110.(1[0-8][0-9]). [NC]
RewriteRule .* - [F,L]
Options -Indexes

 

[The post Who’s Blocked by Bad Guys? has been first published on /dev/random]

August 20, 2017

FOSDEM 2018 will take place at ULB Campus Solbosch on Saturday 3 and Sunday 4 February 2018. Further details and calls for participation will be announced in the coming days and weeks.

August 18, 2017

In the .NET XAML world, you have the ICommand, the CompositeCommand and the DelegateCommand. You use these commands to in a declarative way bind them as properties to XAML components like menu items and buttons. You can find an excellent book on this titled Prism 5.0 for WPF.

The ICommand defines two things: a canExecute property and an execute() method. The CompositeCommand allows you to combine multiple commands together, the DelegateCommand makes it possible to pass two delegates (functors or lambda’s); one for the canExecute evaluation and one for the execute() method.

The idea here is that you want to make it possible to put said commands in a ViewModel and then data bind them to your View (so in QML that’s with Q_INVOKABLE and Q_PROPERTY). Meaning that the action of the component in the view results in execute() being called, and the component in the view being enabled or not is bound to the canExecute bool property.

In QML that of course corresponds to a ViewModel.cpp for a View.qml. Meanwhile you also want to make it possible to in a declarative way use certain commands in the View.qml without involving the ViewModel.cpp.

So I tried making exactly that. I’ve placed it on github in a project I plan to use more often to collect MVVM techniques I come up with. And in this article I’ll explain how and what. I’ll stick to the header files and the QML file.

We start with defining a AbstractCommand interface. This is very much like .NET’s ICommand, of course:

#include <QObject>

class AbstractCommand : public QObject {
    Q_OBJECT
    Q_PROPERTY(bool canExecute READ canExecute NOTIFY canExecuteChanged)
public:
    AbstractCommand(QObject *parent = 0):QObject(parent){}
    Q_INVOKABLE virtual void execute() = 0;
    virtual bool canExecute() const = 0;
signals:
    void canExecuteChanged(bool canExecute);
};

We will also make a command that is very easy to use in QML, the EmitCommand:

#include <MVVM/Commands/AbstractCommand.h>

class EmitCommand : public AbstractCommand
{
    Q_OBJECT
    Q_PROPERTY(bool canExecute READ canExecute WRITE setCanExecute NOTIFY privateCanExecuteChanged)
public:
    EmitCommand(QObject *parent=0):AbstractCommand(parent){}

    void execute() Q_DECL_OVERRIDE;
    bool canExecute() const Q_DECL_OVERRIDE;
public slots:
    void setCanExecute(bool canExecute);
signals:
    void executes();
    void privateCanExecuteChanged();
private:
    bool canExe = false;
};

We make a command that allows us to combine multiple commands together as one. This is the equivalent of .NET’s CompositeCommand, here you have our own:

#include <QSharedPointer>
#include <QQmlListProperty>

#include <MVVM/Commands/AbstractCommand.h>
#include <MVVM/Commands/ListCommand.h>

class CompositeCommand : public AbstractCommand {
    Q_OBJECT

    Q_PROPERTY(QQmlListProperty<AbstractCommand> commands READ commands NOTIFY commandsChanged )
    Q_CLASSINFO("DefaultProperty", "commands")
public:
    CompositeCommand(QObject *parent = 0):AbstractCommand (parent) {}
    CompositeCommand(QList<QSharedPointer<AbstractCommand> > cmds, QObject *parent=0);
    ~CompositeCommand();
    void execute() Q_DECL_OVERRIDE;
    bool canExecute() const Q_DECL_OVERRIDE;
    void remove(const QSharedPointer<AbstractCommand> &cmd);
    void add(const QSharedPointer<AbstractCommand> &cmd);

    void add(AbstractCommand *cmd);
    void clearCommands();
    QQmlListProperty<AbstractCommand> commands();

signals:
    void commandsChanged();
private slots:
    void onCanExecuteChanged(bool canExecute);
private:
    QList<QSharedPointer<AbstractCommand> > cmds;
    static void appendCommand(QQmlListProperty<AbstractCommand> *lst, AbstractCommand *cmd);
    static AbstractCommand* command(QQmlListProperty<AbstractCommand> *lst, int idx);
    static void clearCommands(QQmlListProperty<AbstractCommand> *lst);
    static int commandCount(QQmlListProperty<AbstractCommand> *lst);
};

We also make a command that looks a lot like ListElement in QML’s ListModel:

#include <MVVM/Commands/AbstractCommand.h>

class ListCommand : public AbstractCommand
{
    Q_OBJECT
    Q_PROPERTY(AbstractCommand *command READ command WRITE setCommand NOTIFY commandChanged)
    Q_PROPERTY(QString text READ text WRITE setText NOTIFY textChanged)
public:
    ListCommand(QObject *parent = 0):AbstractCommand(parent){}
    void execute() Q_DECL_OVERRIDE;
    bool canExecute() const Q_DECL_OVERRIDE;
    AbstractCommand* command() const;
    void setCommand(AbstractCommand *newCommand);
    void setCommand(const QSharedPointer<AbstractCommand> &newCommand);
    QString text() const;
    void setText(const QString &newValue);
signals:
    void commandChanged();
    void textChanged();
private:
    QSharedPointer<AbstractCommand> cmd;
    QString txt;
};

Let’s now also make the equivalent for QML’s ListModel, CommandListModel:

#include <QObject>
#include <QQmlListProperty>

#include <MVVM/Commands/ListCommand.h>

class CommandListModel:public QObject {
    Q_OBJECT
    Q_PROPERTY(QQmlListProperty<ListCommand> commands READ commands NOTIFY commandsChanged )
    Q_CLASSINFO("DefaultProperty", "commands")
public:
    CommandListModel(QObject *parent = 0):QObject(parent){}
    void clearCommands();
    int commandCount() const;
    QQmlListProperty<ListCommand> commands();
    void appendCommand(ListCommand *command);
    ListCommand* command(int idx) const;
signals:
    void commandsChanged();
private:
    static void appendCommand(QQmlListProperty<ListCommand> *lst, ListCommand *cmd);
    static ListCommand* command(QQmlListProperty<ListCommand> *lst, int idx);
    static void clearCommands(QQmlListProperty<ListCommand> *lst);
    static int commandCount(QQmlListProperty<ListCommand> *lst);

    QList<ListCommand* > cmds;
};

Okay, let’s now put all this together in a simple example QML:

import QtQuick 2.3
import QtQuick.Window 2.0
import QtQuick.Controls 1.2

import be.codeminded.mvvm 1.0

import Example 1.0 as A

Window {
    width: 360
    height: 360
    visible: true

    ListView {
        id: listView
        anchors.fill: parent

        delegate: Item {
            height: 20
            width: listView.width
            MouseArea {
                anchors.fill: parent
                onClicked: if (modelData.canExecute) modelData.execute()
            }
            Text {
                anchors.fill: parent
                text: modelData.text
                color: modelData.canExecute ? "black" : "grey"
            }
        }

        model: comsModel.commands

        property bool combineCanExecute: false

        CommandListModel {
            id: comsModel

            ListCommand {
                text: "C++ Lambda command"
                command:  A.LambdaCommand
            }

            ListCommand {
                text: "Enable combined"
                command: EmitCommand {
                    onExecutes: { console.warn( "Hello1");
                        listView.combineCanExecute=true; }
                    canExecute: true
                }
            }

            ListCommand {
                text: "Disable combined"
                command: EmitCommand {
                    onExecutes: { console.warn( "Hello2");
                        listView.combineCanExecute=false; }
                    canExecute: true
                }
            }

            ListCommand {
                text: "Combined emit commands"
                command: CompositeCommand {
                    EmitCommand {
                        onExecutes: console.warn( "Emit command 1");
                        canExecute: listView.combineCanExecute
                    }
                    EmitCommand {
                        onExecutes: console.warn( "Emit command 2");
                        canExecute: listView.combineCanExecute
                    }
                }
            }
        }
    }
}

I made a task-bug for this on Qt, here.

The post MariaDB: JSON datatype supported as of 10.2 appeared first on ma.ttias.be.

Let's say you have the following CREATE TABLE statement you wanted to run on a MariaDB instance.

CREATE TABLE `test` (
  `id` int unsigned not null auto_increment primary key,
  `value` json null
) default character set utf8mb4 collate utf8mb4_unicode_ci

You might be greeted with this error message;

SQLSTATE[42000]: Syntax error or access violation: 1064 You have an error in your SQL
syntax; check the manual that corresponds to your MariaDB server version for the right
syntax to use near 'json null at line 1

Syntax-wise all is good, but the JSON data type is actually pretty new, and it's only "supported" (these parentheses will become clear in a moment) as of MariaDB 10.2.

First of all: make sure you run the latest MariaDB.

MariaDB> select @@version;
+-----------------+
| @@version       |
+-----------------+
| 10.0.31-MariaDB |
+-----------------+

If, like me, you were on an older release, upgrade that until you're on MariaDB 10.2.

MariaDB> select @@version;
+----------------+
| @@version      |
+----------------+
| 10.2.7-MariaDB |
+----------------+

Then, be aware that MariaDB's JSON implementation is slightly different than MySQL's, as explained in MDEV-9144.

JSON data type directly contradicts SQL standard, that says, that JSON_* functions take a string as an argument.

Also, speed-wise MariaDB does not need binary JSON, according to our benchmarks, our JSON parser is as fast on text JSON as MySQL on binary JSON.
[...]
We'll add JSON "type" for MySQL compatibility, though.

And as a final remark;

added JSON as an alias for TEXT

So behind the scenes, a JSON data type is actually a TEXT data type. But at least those CREATE TABLE queries will actually work.

That same CREATE TABLE statement above gets translated to this in MariaDB;

MariaDB> SHOW CREATE TABLE test;

CREATE TABLE `test` (
  `id` int(10) unsigned NOT NULL AUTO_INCREMENT,
  `value` text COLLATE utf8mb4_unicode_ci DEFAULT NULL,
  PRIMARY KEY (`id`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci

Note the TEXT datatype for the 'value' column, that I specified as JSON.

The post MariaDB: JSON datatype supported as of 10.2 appeared first on ma.ttias.be.

August 17, 2017

I published the following diary on isc.sans.org: “Maldoc with auto-updated link“.

Yesterday, while hunting, I found another malicious document that (ab)used a Microsoft Word feature: auto-update of links. This feature is enabled by default for any newly created document (that was the case for my Word 2016 version). If you add links to external resources like URLs, Word will automatically update them without any warning or prompt… [Read more]

[The post [SANS ISC] Maldoc with auto-updated link has been first published on /dev/random]

Welcome to WordPress. This is your first post. Edit or delete it, then start writing!

I hope by now we all integrate with third party tooling when it comes to validating code changes and enforcing conventions and good practice. But what do you do when you can’t find what you’re looking for? That’s exactly where I was when one of my team members asked if it’s possible to block merging pull requests in GitHub when there are still fixup or squash commits to be found (this all boils down to our branching strategy, more on that in a future blog post).

We want to proceed with safety, which means no-one should have to worry about merging something which isn’t ready. People have to be able to focus on being awesome and everything in the development process should facilitate that. When we build gated check ins, we want to cover everything we can, leaving no room for mistakes. As we all fail at times, we have to automate wherever we can.

Luckily for us, GitHub provides a rich API that allows to interact with basically everything. Creating an automated review tool starts by setting up a small API where GitHub can post events to. Now, I’ve tried numerous languages and frameworks, and when it comes to quickly creating a web API, I always go to Express in nodejs. You can hate on Javascript all you want, but even in a swamp there are flowers to be found.

Thanks to an active community, we can make use of two modules to help us out. express-x-hub to assist in validate requests, and octonode to interact with the GitHub API. We start out simple by creating an endpoint to integrate with GitHub and test the API. I created a gist that contains all the code needed to follow along. Let’s go over the files and their responsibilities.

As we’re dealing with a nodejs project, we’ve got our obligatory package.json file containing the project description and dependencies. I prefer yarn over npm, so let’s meet these dependencies using yarn install to download and install these locally. There are two other files to be found, app.js contains the logic to run our express API and .env holds the environment variables we do not wish to expose in the source code (you want to add this to your .gitignore file). For your enjoyment, I also included the launch.json debug settings for Visual Studio Code.

Verify if you can run everything locally before connecting this to the web. Open the project in Visual Studio Code, open app.js and press F5. Great, you’ve got a little API running in debug mode. Navigating to localhost:3000 should display a hello world in your browser (rejoice). Of course, we aren’t interacting with GitHub yet. Thanks to the marvels of today’s development tools, we can use ngrok to expose the API running on our machine. Install ngrok, and expose the API running on port 3000 using ngrok http 3000 in your preferred shell environment.

Next up, we have to create a webhook in the GitHub repository you wish to interact with. Go to Settings/Webhooks/Add webhook to create a new one. Enter https://<GUID>.ngrok.io/github-hook using the specific endpoint ngrok provided you with, just make sure to use https. Set Content type to application/json and enter a random secret (save this value for later). Enable Let me select individual events, select Pull request and remove the Push event. Pressing Add webhook immediately triggers a POST to our newly created API which should then print the received data. You can see the data GitHub sends by opening a webhook’s details and scroll to the bottom. There’s a section Recent Deliveries that contains the payloads and response data. For our newly received request, you should see a response status 200 and the same payload printed by the API running on our machine.

To talk to GitHub, we will use a personal access token. Whether you’re setting this up for a personal repository or not, you might want to go for a token from a bot or regular user. On GitHub, navigate to Your personal settings/Personal access tokens/Generate new token, add a description and select the high level repo scope. Generate the token and copy the key. Go back to the project in Visual Studio Code and edit the .env file using your newly generated token and the previously generated secret when setting up the webhook (you can always edit the hook in case you forgot this).

XHUB_SECRET=your_secret
GITHUB_API_KEY=your_personal_access_token

There’s one file in the code sample we didn’t cover yet. github.js contains a function using octonode to interact with GitHub and set a commit status. We will use this to set a status on the last commit in a pull request. Change the app.post('/github-hook', function(req, res) function in app.js to look like this and restart the API using F5.

app.post('/github-hook', function(req, res) {
    var isValid = req.isXHubValid();
    if (!isValid) {
        res.status(404).end();
        return;
    }
    res.status(200).end();
    github.setCommitStatus(req.body.pull_request.head.repo.full_name,
        req.body.pull_request.head.sha,
        'failure',
        'Oh no, this commit looks bad!',
        'github-tools/test');
});

Let’s have a look at this function. First, the secret you selected when setting up the webhook serves as validation for the request. This to make sure the received request is indeed coming from our repository. If it can’t validate the request, we’ll simply return a 404 because we’re not really here. Once the request passes validation, we immediately return a 200 as the processing of the pull request might take longer than the time it takes GitHub to mark the request as timeout. Finally, we set a hardcoded commit status on the HEAD of the pull request.

GitHub provides 4 types of commit statuses: pending, success, error, or failure. Depending on your use-case, you might want to mark commits pending before assigning the final status. And make sure to only mark them as error when your API code failed to validate due to a bug.

Now, when you push a branch in the repository you added the webhook to and create a pull request, you will see the data being posted to your API. If you did everything according to the sample, the API will immediately mark the commit as failure and GitHub will display the message Oh no, this commit looks bad!. The cool thing is, now you can mark this check as required in the repository’s settings to enforce the rule and not allow anyone to bypass this check. Navigate to Branches/Protected branches/<branch>/edit/Require status checks to pass before merging/<select github-tools/test>.

Using this sample, you can extend it towards whatever use-cases your team could benefit from. You can find the implementation our team currently uses in my github-tools repository. We currently have 2 use-cases, one is to check for any fixup or squash commits in the pull requests, the other one will look for changes in requirements.txt files and make sure everyone neatly follows PEP440 rules and properly sets dependencies. As the possibilities are endless, I can’t wait to see what you’ll come up with, so make sure to let me know in the comments or on Twitter!

August 16, 2017

I published the following diary on isc.sans.org: “Analysis of a Paypal phishing kit“.

They are plenty of phishing kits in the wild that try to lure victims to provide their credentials. Services like Paypal are nice targets and we can find new fake pages almost daily. Sometimes, the web server isn’t properly configured and the source code is publicly available. A few days ago, I was lucky to find a ZIP archive containing a very nice phishing kit targeting Paypal. I took some time to have a look at it… [Read more]

[The post [SANS ISC] Analysis of a Paypal phishing kit has been first published on /dev/random]

August 15, 2017

Dri es
I recently was able to obtain the domain name dri.es so I decided to make the switch from buytaert.net to dri.es. I made the switch because my first name is a lot easier to remember and pronounce than my last name. It's bittersweet because I've been blogging on buytaert.net for almost 12 years now. The plan is to stick with dri.es for the rest of the blog's life so it's worth the change. Old links to buytaert.net will automatically be redirected, but if you can, please update your RSS feeds and other links you might have to my website.
Dri es
I recently was able to obtain the domain name dri.es so I decided to make the switch from buytaert.net to dri.es. I made the switch because my first name is a lot easier to remember and pronounce than my last name. It's bittersweet because I've been blogging on buytaert.net for almost 12 years now. The plan is to stick with dri.es for the rest of the blog's life so it's worth the change. Old links to buytaert.net will automatically be redirected, but if you can, please update your RSS feeds and other links you might have to my website.

August 13, 2017

Dans ce billet, j’explore le futur de la nutrition en testant différents repas en poudre, comme je l’avais déjà fait il y’a deux ans.

Update : ce billet a été mis à jour le 11 septembre 2017 pour ajouter le test de Saturo, préciser la compatibilité avec les régimes végans et pour ajouter des liens de parrainage.

Pour produire l’énergie nécessaire à la vie, nous n’avons besoin que de deux choses : du carburant et du comburant. Vu sous cet angle, tout notre système digestif ne sert qu’à une seule et unique chose : extraire du carburant de notre environnement en rejetant l’immense majorité qui n’est pas utilisable. Le comburant, lui, est fourni par le système respiratoire.

Toute cette complexité organique, toute cette énergie, toutes ces sources potentielles de maladies et de complications pour une seule et unique chose : extraire de tout ce qui nous entoure du carbone (et quelques autres composants) que l’on pourra ensuite combiner à de l’oxygène pour produire de l’énergie.

Entre parenthèse, cela signifie aussi que si nous sommes trop gros, la seule et unique manière de nous débarrasser du carbone excédentaire est de… respirer. En effet, le CO2 que nous expirons est la seule porte de sortie pour le carbone de notre corps, avec l’urine qui en contient également une toute petite quantité.

Le système digestif étant extrêmement énergivore, un comble vu que son rôle est d’obtenir de l’énergie, l’être humain inventa la cuisine. Les recettes permirent de sélectionner les aliments les plus nourrissants tandis que la cuisson, rendue possible par le feu, facilita la digestion.

Depuis, si les recettes de cuisine sont brandies comme un étendard culturel, force est de constater que nous mangeons majoritairement des ersatz industriels des recettes originelles. Les industriels ont compris comment tromper nos réflexes pour nous faire ingérer de la nourriture bon marché. Le sucre, initialement un indicateur naturel d’un fruit mûr contenant de bonnes vitamines, a été isolé pour être saupoudré dans à peu près tout, nous rendant accros à des produits peu nourrissants voire franchement nocifs.

Quelle sera donc la prochaine évolution en termes de nutrition ? S’il n’est pas question de gaspiller une occasion de manger un bon repas, je suis persuadé que la malbouffe industrielle et le sandwich de midi peuvent avantageusement être remplacés par de la nourriture spécialement conçue pour apporter ce dont le corps à besoin le plus efficacement possible. C’est exactement l’objectif du Soylent, qui a donné naissance à de nombreux clones européens dont je vous avais parlé il y’a deux ans en vous posant la question « Est-il encore nécessaire de manger ? ».

Or, en deux ans, les choses ont bien changé. Des alternatives françaises ont vu le jour et les produits ont gagné en qualité. Je vous propose un petit tour d’horizon des différents repas en poudre que j’ai testé.

Vitaline

Vitaline, c’est le produit santé de cette comparaison. Composé d’ingrédients essentiellement bio, Vitaline cherche avant tout la qualité. D’ailleurs, les premières versions étaient peu nourrissantes et au goût assez fade. La dernière version a grandement amélioré ces aspects même si on est encore limité à 3 goûts (pour moi 2 car je n’aime pas du tout le goût amande alors que je raffole pourtant du massepain).

Autre particularité : Vitaline est le seul des produits testés qui périme assez vite. La poudre devient immangeable après quelques semaines de stockage là où les autres restent identiques durant plusieurs mois, voire années. Peut-être est-ce le prix à payer pour avoir des composants bio et moins de conservateurs mais ce n’est pas très pratique.

À 4€ le repas, je conseille Vitaline à ceux pour qui la santé et le bio passent avant le goût, ce dernier pouvant être un peu amer. J’apprécie aussi énormément les sachets individuels, bien plus pratiques que les gros sachets de 3 repas. Avec mon lien de parrainage, vous aurez 20% de réduction sur le pack découverte et je recevrai 10€ sur ma prochaine commande.

Note : Vitaline m’a spontanément offert deux coffrets de test suite à la lecture de mon article d’il y’a deux ans.

Smeal

Autre alternative française, Smeal ne se démarque pas spécialement. Les goûts sont bons (parfois de manière surprenante, comme Speculoos) mais fort sucrés et fort écœurants. À 3€ le repas, je l’ai plutôt perçu comme une alternative bon marché qui remplit son office : on n’a plus faim pendant plusieurs heures après un Smeal.

Soulignons la poudre goût « légumes du jardin ». Une véritable innovation qui permet de sortir de l’aspect essentiellement sucré de ces repas.

Note : suite à ma demande, j’ai reçu un pack de test gratuit de Smeal.

Feed

Toujours en France, Feed se démarque par son aspect design et pratique. Plutôt que les traditionnels sachets, Feed propose des bouteilles en plastique pré-remplies de poudre. Une innovation d’ailleurs reprise par Vitaline.

Dans une première version, la poudre formait de tels grumeaux que je n’ai jamais réussi à la mélanger correctement dans les bouteilles. C’est de plus particulièrement peu écologique.  Feed a amélioré sa recette pour en plus avoir de problème de grumeaux. Je n’ai cependant pas été complètement convaincu par l’expérience ni par le goût, même si c’est fort pratique de pouvoir se passer d’un shaker à nettoyer lors de journées nomades.

Notons que, comme Smeal, Feed se diversifie dans les goûts salés. Feed propose également des barres repas. Ces barres sont nourrissantes sans être écœurantes et proposent des goûts salés. Bref, de Feed, je retiens essentiellement les barres qui sont particulièrement bonnes et nourrissantes.

Notons que Feed ne contient pas de lactose et est donc adapté à un régime vegan. Si vous commandez pour 50€ chez Feed en utilisant ce lien, je recevrai une ristourne de 10€.

Note : suite à ma demande, j’ai reçu un pack de test gratuit de Feed. Un deuxième pack m’a été envoyé suite à ma critique concernant les grumeaux et, effectivement, ce problème avait disparu.

Queal

Déjà testé il y’a deux ans, Queal, produit hollandais, oriente désormais son marketing sur la performance, physique et intellectuelle. Pour le gag, il faut noter que leur nouveau shaker est le moins performant du marché, à la limite de l’inutilisable avec un bouchon qui se détache et qui est inlavable.

Mais force est de constater que leur poudre reste pour moi la plus digeste, avec une pléthore de goûts dont certains sont délicieux. À 2,5€ le repas, Queal reste un maître achat.

Queal tente de se diversifier avec des barres repas, les Wundrbars, qui sont absolument infectes mais nourrissantes (elles gardent toutes une trace d’amertume très prononcées).

Autre innovation, Queal propose la poudre « boost », un supplément nootropique permettant d’améliorer la mémoire et la concentration. L’effet sur la mémoire de certains composants du Boost serait démontré scientifiquement.

Est-ce que ça fonctionne ? J’ai l’impression que les matins où je rajoute du Boost à mon Queal, je suis plus apaisé et légèrement plus concentré que d’habitude. Je me sens moins grognon et moins enclin à procrastiner. Effet placebo ? C’est fort probable. À 60€ le pot de boost, je n’ai pas envie de gâcher l’effet en glandant sur Facebook !

Note : Queal m’a spontanément contacté pour m’offrir un coffret de test suite à mon article d’il y’a deux ans.

Ambronite

Produit d’ultra luxe, à plus de 8€ le repas, Ambronite se démarque grandement par sa composition.

Là où tous les autres produits sont essentiellement de la protéine de lait avec des suppléments et des arômes, Ambronite est un réel mélange de fruits et légumes secs réduits en poudre. Toutes les vitamines et les minéraux sont issues de produits naturels, les protéines étant essentiellement fournies par de l’avoine.

Il en résulte une espèce de soupe verte avec des arrières goûts sucrés de fruits. Le fait qu’il n’y aie pas de protéine de lait rend Ambronite beaucoup plus digeste et moins écœurant. Ambronite se diversifie désormais avec des goûts mais aucun ne m’a convaincu. Tant qu’à choisir Ambronite, je conseille nettement l’original.

Le problème d’Ambronite reste avant tout son prix. Cela nous confronte à une question intéressante. Si payer 3/4€ pour être nourri en sautant en repas semble « rentable », suis-je prêt à payer plus du double pour un repas qui ne m’apportera aucun plaisir gustatif et qui sera ingéré en quelques secondes ?

Grâce à ce lien, vous pouvez recevoir un paquet de test Ambronite gratuit contenant 4 petits paquets de goût différent. Chaque paquet est l’équivalent d’une collation ou d’un tiers de repas. Vous devez néanmoins payer les frais de port (5,90€).

Note : Je n’ai aucun avantage si vous utilisez ce lien. Tout au plus ai-je reçu ce paquet de test sans payer les frais de port.

Saturo

Saturo pousse le concept du repas en bouteille jusqu’à déjà remplir et mélanger. Il ne reste donc plus qu’à ouvrir et boire.

Et, de manière surprenante, ça fonctionne très bien. Saturo a bon goût et est idéal avant le sport. À 3€ la bouteille, c’est un excellent rapport qualité-prix même si une bouteille n’est pas exactement un repas. Au niveau petit-déjeuner efficace, je recommande Saturo. Notons également que Saturo est particulièrement digeste, ne contient pas de lactose et est vegan friendly !

Je pense qu’en commandant du Saturo via ce lien, vous pouvez obtenir une réduction et moi-aussi (mais je ne suis pas sûr, j’ai pas bien compris le parrainage).

Note : Saturo m’a proposé spontanément un paquet de test suite à la lecture de cet article.

Conclusion du test

Au niveau des marques, si ce n’était son prix prohibitif, je pense que je consommerais essentiellement de l’Ambronite, à la fois bon, efficace et excellent pour la santé. C’est également celui que je recommande sans aucune hésitation à mes enfants. Pour plus de facilité, je garde également toujours du Saturo au frigo et quelques barres de Feed.

Dans une gamme de prix correcte, j’apprécie la démarche de Vitaline, qui fait passer la santé et les aspects scientifiques de son produit avant le goût et le marketing. Malgré la composition essentiellement basée sur la protéine de lait, l’attention portée à l’utilisation d’ingrédients bio me rendent également confortable avec le fait d’offrir Vitaline à mes enfants même si le goût ne le rend pas très attractif.

Mais pour l’usage quotidien, Queal reste une valeur sûre, au goût « facile » qui plaira à tout le monde. Pour les enfants, je me rassure en me disant que ça ne peut pas être pire qu’un hamburger mais je ne pousse pas à l’utilisation trop fréquente de Queal. Je reste également partagé sur le supplément Boost. C’est soit absolument génial, soit une arnaque complète. Je n’arrive pas à me décider.

Une évolution rapide et souhaitable

En deux ans, la qualité des repas en poudre a monté de plusieurs crans. De nombreux produits sont apparus et, parfois, ont disparu aussitôt. C’est d’ailleurs un peu difficile pour le consommateur de s’y retrouver.

Mais force est de constater que ce genre de produits s’installe durablement. En deux ans, il m’est arrivé de manger essentiellement des repas en poudre pendant plusieurs jours et je me sentais particulièrement plein d’énergie. Les selles se font également plus légères. Je ne sais personnellement plus me passer d’un repas en poudre avant une longue randonnée à vélo. Même au niveau du travail intellectuel, je sais qu’un repas en poudre favorise ma concentration par rapport à tout autre repas.

Je pense que l’innovation principale sera dans l’abandon progressif de la protéine de lait et la démocratisation des produits de très haute qualité, comme Ambronite. Une attention particulière sera de plus en plus portée au bilan carbone du repas, à l’absence de produits indésirables (pesticides, sucres raffinés). En parallèle, je prédis l’apparition de produits très bon marché (moins de 1€ le repas) mais à la qualité bien moindre.

Loin de rester des alternatives aux repas, ces poudres en deviendront des composants, avec la popularisation de recettes utilisant les poudres pour les mélanger à d’autres ingrédients. Il deviendra socialement acceptable voire normal de consommer des repas en poudre là où, aujourd’hui, on me regarde encore souvent comme un extra-terrestre.

Et après ?

À plus long terme, je suis convaincu que l’on considérera la manière dont nous nous alimentons aujourd’hui comme préhistorique et morbide. Sans aucune considération pour la valeur nutritive, nous avons en effet tendance à nous laisser diriger par notre goût, notre odorat et notre vue, sens facilement abusés par la publicité, le marketing et les additifs chimiques. Si l’idée d’un repas en poudre en choque certains, il faut peut-être rappeler que nous avons tous passé les premiers mois de notre vie nourris par une source de nourriture unique (que ce soit en poudre ou à travers l’allaitement maternel).

Sur le principe de l’imprimante 3D, nous aurons alors dans notre cuisine un shaker qui mélangera en direct les ingrédients en se basant sur notre envie du moment pour le goût et sur les données de bio-capteurs pour la valeur nutritive nécessaire à notre organisme.

Les imprimantes 3D les plus sophistiquées pourront reproduire le goût et la consistance de la plupart des aliments connus, y compris la viande. Le prix et l’encombrement réserveront néanmoins dans un premier temps ces appareils aux restaurants. Il sera possible de commander un steak saignant vegan, riche en vitamines et pauvre en graisse.

L’époque où nous ingérions des graisses saturées issues d’animaux morts en buvant des sodas nous semblera probablement particulièrement barbare. Tout comme il ne nous viendrait pas à l’esprit aujourd’hui de tuer un animal et d’en arracher la chair encore chaude avec les dents, le visage barbouillé de sang.

Remarque importante : ce blog est financé par ceux d’entre vous qui m’offrent un prix libre pour chaque série de 5 billets. Cela se passe sur Tipeee ou Patreon et je ne vous remercierai jamais assez pour votre contribution. Je considère ce billet comme ayant été financé par la réception d’échantillons Vitaline, Smeal et Feed. Il n’est pas payant et ne compte pas dans la prochaine série de 5.

Photo par Brian Brodeur.

Vous avez aimé votre lecture ? Soutenez l’auteur sur Tipeee, Patreon, Paypal ou Liberapay. Même un don symbolique fait toute la différence ! Retrouvons-nous ensuite sur Facebook, Medium, Twitter ou Mastodon.

Ce texte est publié sous la licence CC-By BE.

August 11, 2017

So I wanted to replace a site’s (*) main title which required some fancy font (Courgette!) to be downloaded, by an SVG image.

Here’s what I did;

  1. make a screenshot of the node in Firefox Developer Tools -> Inspector -> select node -> click “screenshot node” in context menu
  2. convert the png-file into svg at http://pngtosvg.com/ result being a 6.93KB file.
  3. optimize the svg at http://petercollingridge.appspot.com/svg-optimiser resulting in a 3.1KB file (see above image) which remains crispy at whatever size.
  4. added the SVG as background image (not inline though, might do that next) and set “visibility” of the logo->a->h3 (which has the title in it as text) to “hidden”
  5. ticked Autoptimize’s “remove Google Fonts”-option (which also removed a slew of other unwanted requests for fonts)

(*) The site is my wife’s boeken-jagers.be which is an offspring of her successful “De Boekenjagers” Facebook group where people hide books for others to find (hunt) and share info about that. 27 000 members and counting, proud of my Veerleken!

August 10, 2017

As heard on an older Gilles Peterson WorldWide-show I listened to on the train this morning.

YouTube Video
Watch this video on YouTube.

About Cossi Anatz:

Comment ça va ? Cossi Annatz in french occitan means “How are you doing ?” I’m doing well when I listen to this beautiful free/spiritual jazz album recorded and produced by Michel Marre in the South of France and released on Vendémiaire (Jef Gilson’s label) with a similar style to Francois Tusques Intercommunal. Jazz Afro-occitan is written on the cover and it’s clearly the case when you put the needle on wax you instantly feel the North African, Arabic & Mediterranean influences, that are quite obvious here, especially in the brass & percussions. Some brilliant tracks like the beautiful deep jazz Al Coltrane or the monster spiritual oriental jazz Aden Arabie with great Fender solo.

(Source: http://www.diggersdigest.com/shop/product/jazz-afro-occitan-1/)

August 08, 2017

Due to some technical issues, it took a slight bit longer than I'd originally expected; but the first four videos of the currently running DebConf 17 conference are available. Filenames are based on the talk title, so that should be reasonably easy to understand. I will probably add an RSS feed (like we've done for DebConf 16) to that place some time soon as well, but code for that still needs to be written.

Meanwhile, we're a bit behind on the reviewing front, with (currently) 34 talks still needing review. If you're interested in helping out, please join the #debconf-video channel on OFTC and ask what you can do. This is something which you can do from home if you're interested, so don't be shy! We'd be happy for your help.