Dear *,

We plan to organize an small event this year – 2012 – in Belgium on how to build your own social community Platform with Enterprise Open Source Softwares inside your company.

This event should cover the user experiences services like :

• Enterprise Social
• Communication
• Collaboration
• Content Management
• Custom Application Development
• Mobile Applications

So, If you or your company are based in BeNeLux and are interested by this subject, then just let me know and I will arrange a Meeting for you.

BR

Frederic

I'm a big fan of Flattr. But I find it hard to have some statistics about your things that have been flattered.

On my Flattr account, I receive flatts for both my blog and for Getting Things GNOME!. But I want to keep a clear separation. There are multiple persons now involved in GTG and they deserve part of the money (we will use that to buy beers at FOSDEM).

Also, on my own blog, I was interested to know which posts where the more successful, speaking of revenue. I knew that, so far, this post had the most clicks but I had no idea which one received the most money (for the curious, it is that one).

In order to do that, I quickly wrote FlattrStat, a python script. You need to download all the csv files from flatr, put them in a folder then run the script with "python flattrstat.py".

It will outputs the total clicks and revenues for each domain separately and, for each domain, sort all your things from the most successful to the least one.

Ideally, it should download the CSV files automatically and have a nice GUI but I don't really need that. It was for my own needs but I realize that it might be useful to someone else. So, feel free to use it or to contribute, it is under the WTFPL license.

FlattrStat on GitHub

Dear all,

I am going to do a presentation about “Red Hat Enterprise Virtualization 3.0” @ Altimate in Brussels in February 2012.
If your company or you are based in BeNeLux and are interested by this presentation, just let me know and I will try to arrange a Meeting for you.

Ref :  http://www.redhat.com/promo/rhev3

Ref : http://www.altimate.be/brand/17694b441900027b/Red-Hat-JBoss.html

Ref : http://www.altimate.be/local/17694b44190000f9/1e1537a81840686a/Red-Hat-RHEV-3-0-Launch-party-on-the-2nd-of-February-.html

Best Regards

Frederic

Wazaabi est un framework open-source pour faciliter la création d'interfaces graphiques dans des applications de client riche tel que Eclipse RCP. Si vous avez déjà programmé des interfaces graphiques pour ce types d'applications, vous avez pu remarquer à quel point c'est une réelle prise de tête pour faire quelque chose de bien ficelé et de robuste. Je ne vous parle même pas de la maintenance de l'interface graphique lorsque le cahier des charges évolue et qu'il faut faire évoluer l'application. Et il est évidement impensable d'envisager de récupérer des parties de code d'une application pour la recoller dans une autre sous prétexte qu'on veut +/- la même interface graphique.

Tous les développeurs qui ont déjà du développer des interfaces graphiques pour des applications client riches (RCP) ont déjà rencontré ce type de problèmes qui sont des prises de têtes assez importante pour s'en sortir. C'est la raison pour laquelle Wazaabi est là. Nous avons remarqués que de plus en plus d'applications sont développées sur base d'une approche modèle (tel un modèle business) et qu'une interface graphique peut être structurée de la même manière.

L'approche Wazaabi est de rendre le développement de l'interface graphique : simple, facile à maintenir, portable. Wazaabi est depuis peu un projet officiel de la fondation Eclipse. Wazaabi n'est pas basé sur une librairie graphique en particulier. Si vous développez en Swing, en SWT ou même si vous développez sur Android, Wazaabi est une surcouche générique à la librairie que vous utilisez. L'idée est donc de vous permettre facilement, avec une même logique fonctionnelle, d'avoir plusieurs UI en fonction de votre librairie graphique.

Samedi prochain, à l'occasion du Fosdem, je tiendrais un lightning talk pour vous présenter wazaabi en détail. Ca aura lieu samedi 4 février à 18h40 dans l'auditoire Ferrer. Si vous êtes développeur eclipse/java, venez découvrir ce framework qui vous intéressera surement !

En attendant, pour en savoir plus, allez faire un tour sur le site de wazaabi ou sur la page de proposition du projet sur Eclipse.org :

• http://wiki.eclipse.org/Wazaabi/Cre...
• http://wazaabi.org/

Jailbreaking is not a crime, but we shouldn’t take that for granted, because as Bunnie (XBox hacker) writes;

Three years ago, the [U.S.] Copyright Office agreed to create an exemption to the Digital Millennium Copyright Act so that folks could jailbreak their smartphones. But that exemption is about to expire.

Given the fact that the U.S. jailbreaking-scene is an important contributor, I signed the EFF petition which asks the Copyright Office for continued support for jailbreakers;

Being an avid Android-user, jailbreaking permits me to replace heavily customized (and in some ways crippled, think CarrierIQ) vendor-specific versions of Android with clean, crisp, fast and secure after-market “mods” such as CyanogenMod.

You should really sign this as well!

Possibly related twitterless twaddle:

• WordPress stats oddity
• Google loves html5 (in Android 2.0)
• The Magic’s gone, enter Samsung Galaxy S II

aptitude install iscsitarget
aptitude install iscsitarget-dkms (Debian 6)

root@debby6:/etc/iet# cat ietd.conf
Target iqn.2010-02.be.linux-training:storage.lun1
IncomingUser isuser hunter2
OutgoingUser
Lun 0 Path=/iscsi/lun1.img,Type=fileio
Alias LUN1

dd if=/dev/zero of=/iscsi/lun1.img count= bs=

root@debby6:/etc/iet# cat initiators.allow
iqn.2010-02.be.linux-training:storage.lun1

root@debby6:/etc/iet# cat /proc/net/iet/session
tid:1 name:iqn.2010-02.be.linux-training:storage.lun1

aptitude install open-iscsi

root@ubu1104:/etc/iscsi# iscsiadm -m discovery -t st -p debby6
192.168.1.31:3260,1 iqn.2010-02.be.linux-training:storage.lun2

root@ubu1104:/etc/iscsi# iscsiadm -m node --targetname "iqn.2010-02.be.linux-training:storage.lun1" --portal "debby6:3260" --op=update --name node.session.auth.authmethod --value=CHAP
root@ubu1104:/etc/iscsi# iscsiadm -m node --targetname "iqn.2010-02.be.linux-training:storage.lun1" --portal "debby6:3260" --op=update --name node.session.auth.username --value=isuser
root@ubu1104:/etc/iscsi# iscsiadm -m node --targetname "iqn.2010-02.be.linux-training:storage.lun1" --portal "debby6:3260" --op=update --name node.session.auth.password --value=hunter2

I’m in the quiet town of Malaga these three days to attend the GStreamer hackfest. The goal is to port applications over to the 0.11 API which will eventually be 1.0 There’s about 18 people here, which is a good number for a hackfest.

The goal for me is to figure out everything that needs to be done to have Flumotion working with GStreamer 0.11. It looks like there is more work than expected, since some of the things we rely on haven’t been ported successfully.

Luckily back in the day we spent quite a bit of time to layer parts as best as possible so they don’t depend too much on each other. Essentially, Flumotion adds a layer on top of GStreamer where GStreamer pipelines can be run in different processes and on different machines, and be connected to each other over the network. To that end, the essential communication between elements is abstracted and wrapped inside a data protocol, so that raw bytes can be transferred from one process to another, and the other end ends up receiving those same GStreamer buffers and events.

First up, there is the GStreamer Data protocol. Its job is to serialize buffers and events into a byte stream.

Second, there is the concept of streamheaders (which is related to the DELTA_UNIT flag in GStreamer). These are buffers that always need to be send at the beginning of a new stream to be able to interpret the buffers coming after it. In 0.10, that meant that at least a GDP version of the caps needed to be in the streamheader (because the other side cannot interpret a running stream without its caps), and in more recent versions a new-segment event. These streamheaders are analogous to the new sticky event concept in 0.11 – some events, like CAPS and TAG and SEGMENT are now sticky to the pad, which means that a new element connected to that pad will always see those events to make sense of the new data it’s getting.

Third, the actual network communication is done using the multifdsink element (and an fdsrc element on the other side). This element just receives incoming buffers, keeps them on a global buffer list, and sends all of them to the various clients added to it by file descriptor. It understands about streamheaders, and makes sure clients get the right ones for wherever they end up in the buffer list. It manages the buffers, the speed of clients, the bursting behaviour, … It doesn’t require GDP at all to work – Flumotion uses this element to stream Ogg, mp3, asf, flv, webm, … to the outside world. But to send GStreamer buffers, it’s as simple as adding a gdppay before multifdsink, and a gdpdepay after fdsrc. Also, at the same level, there are tcpserversink/tcpclientsrc and tcpclientsink/tcpserversrc elements that do the same thing over a simple TCP connection.

Fourth, there is an interface between multifdsink/fdsrc and Python. We let Twisted set up the connections, and then steal the file descriptor and hand those off to multifdsink and fdsrc. This makes it very easy to set up all sorts of connections (like, say, in SSL, or just pipes) and do things to them before streaming (like, for example, authentication). But by passing the actual file descriptor, we don’t lose any performance – the low-level streaming is still done completely in C. This is a general design principle of Flumotion: use Python and Twisted for setup, teardown, and changes to the system, and where we need a lot of functionality and can sacrifice performance; but use C and GStreamer for the lower-level processor-intensive stuff, the things that happen in steady state, processing the signal.

So, there is work to do in GStreamer 0.11:

• The GStreamer data protocol has not really been ported. gdppay/depay are still there, but don’t entirely work.
• streamheaders in those elements will need adapting to handle sticky events.
• multifdsink was moved to -bad and left with broken unit tests. There is now multisocketsink. But sadly it looks like GSocket isn’t meant to handle pure file descriptors (which we use in our component that records streams to disk for example)
• 0.11 doesn’t have the traditional Python bindings. It uses gobject-introspection instead. That will need a lot of work on the Flumotion side, and ideally we would want to keep the codebase working against both 0.10 and 0.11 as we did for the 0.8->0.10 move. Apparently these days you cannot mix gi-style binding with old-style binding anymore, because they create separate class trees. I assume this also means we need to port the glib2/gtk2 reactors in Twisted to using gobject-introspection.

So, there is a lot of work to be done it looks like. Luckily Andoni arrived today too, so we can share some work.

After discussing with Wim, Tim, and Sebastien, my plan is:

1. create a common base class for multihandlesink, and refactor multisocketsink and multifdsink as subclasses of it
2. create g_value_transform functions to bytestreams for basic objects like Buffers and Events
3. use these transform functions as the basis for a new version of GDP, which we’ll make typefindable this time around
4. support sticky events
5. ignore metadata for now, as it is not mandatory; although in the future we could let gdppay decide which metadata it wants to serialize, so the application can request to do so
6. try multisocketsink as a transport for inside Flumotion and/or for the streaming components.
7. In the latter case, do some stress testing – on our platform, we have pipelines with multifdsink running for months on end without crashing or leaking, sometimes going up to 10000 connections open.
8. Make twisted reactors
9. prototype flumotion-launch with 0.11 code by using gir

That’s probably not going to be finished over this week, but it’s a good start. Last night I started by fixing the unit tests for multifdsink, and now I started refactoring multisocketsink and multifdsink with that. I’ll first try and make unit tests for multisocketsink though, to verify that I’m refactoring properly.

there have been some requests from people attending FOSDEM that would like to go to the Betagroup Coworking to work on Friday and Monday. So they thought that it would be a good idea to host as many partcipants as they could... for
free!

Please see details and signup info here:
http://coworking.betagroup.be/hosting-the-fosdem-participants/ Space is limited and signup will be disabled when they reach their limit.

A new year started and why change good habits? I’m just back from the first OWASP Belgium Chapter meeting of 2012. Here is my quick wrap-up. The organization remains the same, the first few minutes were dedicated to some news from the OWASP organization given by Seba. A survey was organized by the Belgium chapter about the attendees and their expectations. Some results were presented. Most part of the members define their knowledge between “security aware” and “experienced“. Good news, lot of people are ready to participate and submit talk ideas! What’s on their wishlist for 2012? Mobile security, HTML5, SDLC, SAP and more demos/hands-on sessions. One recommendation for this year: more interaction between the chapter meetings (via forums, mailing lists?).

The first speaker was Kris Buytaert about “Devops, secops, devsec or *ops ? A gentle introduction to Devops“. Based on his professional experience, Kris had the opportunity to work as a developer and later to move to the other side: system administration. With his knowledge of both worlds, he is an active member of the “devops” movement. What is “devops“? There are multiple definitions. It started in Belgium with a first meeting in October 2009 and a basic question: “How to go from source to production?“. This is now a growing movement which cannot (still) answer all the issues but which tries to reach different communities. The real problem is to deploy better applications at less costs in less time at lower risks. How? A good principle is CAMS (“Culture, Automation, Measurement and Sharing“). In the old days, deployment of an application could be resumed like this: “Here is a tarball, put it in production now!“. But what about security, dependencies, performance impacts? And such deployments were blocked by other system or security constraints. That’s why people hate sysadmins and security officers!

How to address this problem? By talking about non functional requirements like backup procedures, high-availability, upgrade path, scalability, etc. Try to break the silos between developers and sysadmins. Put people together in the same room. If you successfully address all those issues, you will be able to deploy quickly and safely at any time without the fear of “breaking something“. What’s also important? Make all environments the same, define good set of test data, put configurations in revision control systems. A golden rule is to automate as much as possible. A manual deployment will certainly introduce typo errors in configuration files. There are very powerful tools to implement this (like Puppet). Don’t forget to monitor your applications/systems. Events are organized worldwide to discuss about this topic, have a look at devopdays.org. Some people started to think of a broaden “devops” which could integrate security guys. That’s why other names were introduced like “secops” or “*ops“. If you are interesting on this topic follow Kris and the hashtag #devops on Twitter.

After a short break, the second speaker, Erwin Geirnaert from ZionSecurity, talked about “Hardening web applications against malware attacks“. Erwin presented his personal definition of a “malware“: This is some piece of code which is non destructive, steals information, hijacks credentials and injects fraudulent transactions into applications (like e-banking). Funny remark: they are state of the art devops Indeed they are deployed automatically, they are extensively tested. Some well-known malwares are Zeus or SpyEye. Malwares do not only target financial applications. Some of them attacks social network and, this is the Erwin’s prediction, they could attack cloud applications like SalesForce or Google Apps!

They are really evil pieces of software and can defeat classic protections. They perform fake web content injections, they can bypass two-factors authentication, inject or remove HTML code to hide information (like hiding the stolen money by displaying the old bank account summary). They can bypass VPN and virtual keyboards (by taking screenshots after each mouse click to “see” your passwords). Finally they are always optimized to be undetectable. Example: they introduce timeouts during the data input to prevent a too quick transaction (which could be detected as suspicious). They simulate the human behavior.

To protect against those attack, applications hardening is mandatory. How?

• By reduce attack surface
• By eliminating vulnerabilities
• By mitigating the impact of vulnerabilities

Unfortunately, it’s not always easy to find documentation how to harden a web application. Erwin made some queries on Google to search for “<product> hardening guide“. There is clearly a lack of resources. But hardening must occur at all layers: Not only switches, firewalls, operating systems and webservers configuration must be reviewed but developers have to build and maintain secure code. This is a very good example of devops implementation as explained just before by Kris. Unused features must be disabled, access to console must be limited and admin access, content management systems access must be prohibited from non secure networks.

What about core review? It became more and more difficult to perform a complete code review due to the size and complexity of modern applications. So, why not install a WAF (“Web Application Firewall“) say companies! Good idea but it must be properly implemented. Most WAF’s are left running in non-blocking mode. If you are interested in WAF’s, have a look at the tests performed by Larry Suto in November 2011. A WAF will never make your application bullet-proof!

Ok, you hardened your servers and applications. But what about the weakest link,just after the user of course, the end point or… the browser! Browsers became a nice target for hackers because they are easy to hack. Hackers will always use the easiest way to perform their malicious activities. Browsers must also be hardened by patching them and using sandbox mechanisms. The next step? Our mobile devices. According to Erwin, “2012 will be Android hell!“

To conclude, he gave some malware attack examples against e-banking applications and lot of interesting questions raised about the security of bank operations on mobile devices. My point of view is very simple: Do you really need to perform financial transactions while on the road? Personally when I’m doing my homework (like paying my bills), I do this from home with all the required documents near me.

The next chapter meeting will be scheduled around the 6th or March… See you there!

Today I handed in my thesis for a degree in applied engineering. It feels good to finally have the result of one year of work and absorbing knowledge right there in your hand.

The purpose of the dissertation was to develop a module upon The DataTank in order to allow developers to work with more appropriately structured information from multiple data sources through a single call. The result of this was a language we designed for this thesis called SPECTQL. You can test it over here.

Before all else, a literature study researches the meaning of open data, the leitmotif throughout this dissertation. Besides discussing current legislation on open data and copyright, the organisation of Apps for Ghent, an open data event, have been discussed. The second part of the literature study focuses on the relational model, explains semantics and gives an introduction to the Semantic Web.

You can read the full dissertation in Dutch over here: http://pieter.demo.thedatatank.com/scriptie.php

Pieter

Like at previous DrupalCon's, I'm co-organizing a Core Conversations track at DrupalCon Denver.

The Core Conversations track is a place for people actively working on Drupal or Drupal.org to meet and plan the future of Drupal. Each session is either two 15 minute or one 30 minute presentation, followed by 30 minutes of discussion.

I know a lot of you contribute to Drupal or want to start contributing. If so, Core Conversations are a unique opportunity to present in front of key Drupal contributors, and to make the case for why we need to do more of A or B (e.g. authoring experience improvements, API overhauls, etc.). We need UX conversations, performance conversations, feature conversations, etc. Please share your ideas with the world through Drupal core.

If you have ideas for Drupal core, and you are attending DrupalCon, I suggest that you submit a proposal as soon as possible. The deadline is February 1st so don't wait too long. To get your ideas flowing, here are our conversations from Drupalcon London and Drupal Chicago.

GAAN MIJN KLEINKINDEREN LAST HEBBEN VAN HET KERNAFVAL DAT WE VANDAAG PRODUCEREN ? 

Antwoord van kernvragen hier.

Jullie stellen in het kort:

1. Nucleair afval is maar 1% van alle jaarlijks giftig afval
2. het is goed opspoorbaar
3. wat niet ter zake doende opmerkingen over toekomstige technieken.

Als je weet dat bv. plutonium 239 een halveringstijd heeft van 24000 jaar ga je dat dus voor 240000 jaar van het milieu moeten isoleren.  Hoe kan je garanderen dat de huidige technieken inderdaad garanties bieden voor honderduizenden jaren in de toekomst, welke geologische veranderingen zich ook voordoen?    En gezien we afval blijven bijproduceren komt er toch een punt waar de hoeveelheid afval voor een duurtijd van 10duizenden tot honderduizenden jaren moet worden opgeslagen onrealistisch groot is?

Here are some tricks I use to make my vagrant boxes as small as possible:

Tips:

Booting in single user mode:

I boot in single user mode since it will prevent running services that could output logs. I do this because I zero out all my logs before packaging the box.

Updating:

After updating any package, run yum clean (or the apt equivalent).

When booted in single user mode, don’t forget to start-up your network before updating.

When updating kernels, install the kernel packages, reboot and remove the old kernel packages that are no longer in use. Remember to re-install the VirtualBox add-ons too after a kernel update.

Cleanup:

After doing whatever you need to do with the box, I do some rather nasty stuff to make sure the box uses as little as possible place. If you are using a RAW hard-disks, these might be a bad idea (stuff gets BIG).

• Zero out all remaining unused disk space
• Zero out the swap
• Clear out all log files (I just make them empty, I do NOT delete them)

Script:

(You can find this script – or an older version in /root/tools/cleanup_diskspace.sh on my newer vagrant boxes.)

cat - << EOWARNING 
WARNING: This script will fill up your left over disk space.
 
DO NOT RUN THIS WHEN YOUR VIRTUAL HD IS RAW!!!!!
 
You should NOT do this on a running system. 
This is purely for making vagrant boxes damn small. 
 
Press Ctrl+C within the next 10 seconds if you want to abort!! 
 
EOWARNING 
sleep 10; 
 
echo 'Cleanup log files';
find /var/log -type f | while read f; do echo -ne '' &gt; $f; done;
 
echo 'Whiteout root';
count=`df --sync -kP / | tail -n1  | awk -F ' ' '{print $4}'`;
dd if=/dev/zero of=/tmp/whitespace bs=1024 count=$count;
rm /tmp/whitespace;
 
echo 'Whiteout /boot'
count=`df --sync -kP /boot | tail -n1 | awk -F ' ' '{print $4}'`;
dd if=/dev/zero of=/boot/whitespace bs=1024 count=$count;
rm /boot/whitespace;
 
### Repeat the above for other partitions you have.
 
swappart=`cat /proc/swaps | tail -n1 | awk -F ' ' '{print $1}'`
swapoff $swappart;
dd if=/dev/zero of=$swappart
mkswap $swappart;
swapon $swappart;

Furthermore – about this script – USE IT AT YOUR OWN RISK

The European Parliament will soon vote on ACTA, an international trade agreement that might have a huge impact on the internet as we know it. The YouTube video below explains some of the ramifications (in a propaganda-esque kind of way, but still) and this Wikipedia-article provides some more objective information and contains links to -a lot- more detailed reviews and criticisms.

The bottom-line: if you’re European and you were supporting the great anti-SOPA-protests in the U.S., this is the time to act yourself! Spread the word and sign one or more of these petitions against ACTA:

• Belgium Against ACTA
• Just say “NO” to ACTA
• stopp-acta.info petition

The video:

Possibly related twitterless twaddle:

• Embedding HTML5 YouTube video with WP YouTube Lyte
• Enhanced privacy for embedded YouTube
• Embedding YouTube HTML5-video with newTube

Here is the third batch of interviews with our main track speakers:

• Bdale Garbee (FreedomBox)
• Finne Boonen (healthy communities)
• Guido Trotter (Ganeti)
• Wim Godden (caching and tuning)
• Simon Phipps (OSI)
• Renzo Davoli (Internet of Threads)
• Garrett Serack (CoApp)

read more

The Belgian nuclear forum is launching a very aggressive marketing campaign which supposedly provides answers about nuclear energy to the public.

Reading through the questions and answers seems to confirm my original feeling that this is nothing more than a marketing campaign designed to ease the fears surrounding nuclear enegery instead of actually providing honest answers.  

So, as someone with 4 years of experience in the nuclear energy, I will be asking them some questions and I'll be curious to hear the answers.

The first question that I launched yesterday was roughly translated this:

"Why should we trust the answers provided by kernvragen.be as this site is operated by the nuclear forum, which is a lobby-group of nuclear companies and research institutions?"

I'm still awaiting an answer...
 

Code style checking

Prerequisites:

• You will need a recent enough version of puppet-lint that supports the --log-format flag. Install the gem so that the Jenkins can use it.
• On Jenkins, you will need the Warnings Plugin and the HTML Publisher Plugin.
• Make sure that when checking the module from your VCS, it ends up in WORKSPACE/modules/module_name.

Configuration:

Jenkins

Go to the Configure System page and find the Compiler Warnings settings. Add a new console log parser and call it puppet-lint. I use following configuration for parsing puppet-lint warnings and errors.

Name:

puppet-lint

Regular Expression:

^\s*([^:]+):([0-9]+):([^:]+):([^:]+):\s*(.*)$

Mapping Script:

import hudson.plugins.warnings.parser.Warning
// map regular expression to strings
String fileName = matcher.group(1);
String lineNumber = matcher.group(2);
String kind = matcher.group(3);
String check = matcher.group(4);
String message = matcher.group(5);
// return a Warning.
return new Warning(fileName, Integer.parseInt(lineNumber), check, kind, message);

Example Log Message:

./manifests/params.pp:25:autoloader_layout:error:apache::params not in autoload module layout

Jenkins job configuration

We will add several build steps that will run certain actions on our puppet modules.

1. Check syntax
2. Check style
3. Generate documentation

1. For the syntax check, I use following shell script (add a build step):

for file in $(find . -iname '*.pp'); do
  puppet parser validate --color false --render-as s --modulepath=modules $file || exit 1;
done;

2. For the style check, we use puppet-lint (add another build step):

find . -iname *.pp -exec puppet-lint --log-format "%{path}:%{linenumber}:%{check}:%{KIND}:%{message}" {} \;

3. And for generating documentation:

## Cleanup old docs.
[ -d doc/ ] && rm -rf doc/
## Dummy manifests folder.
! [ -d manifests/ ] && mkdir manifests/
## Generate docs
puppet doc --mode rdoc --manifestdir manifests/ --modulepath ./modules/ --outputdir doc
 
## Fix docs to how I want them, I don't like that the complete workspace is included in all file paths.
if [ -d ${WORKSPACE}/doc/files/${WORKSPACE}/modules ]; then
  mv -v "${WORKSPACE}/doc/files/${WORKSPACE}/modules" "${WORKSPACE}/doc/files/modules"
fi;
grep -l -R ${WORKSPACE} * | while read fname; do sed -i "s@${WORKSPACE}/@/@g" $fname; done;

In your post build section:

• Enable Scan for compiler warnings and select puppet-lint.
• Enable publish HTML reports (use ‘doc‘, ‘index.html‘ and ‘Puppet Docs‘ as values). This will add a link to the Job page linking your generated puppet docs.

That’s about it! Any suggestions / improvements on this are always welcome!

Notes:

• I have some examples/tests setup on my Jenkins instance for testing at http://jenkins.vstone.eu. Since I use this for testing, it might be offline / broken / buggy at times.
• The scripts I use may also require some changes if you are using an older version of puppet. I’m currently using 2.7.x for testing my modules.

Every now and then, you'll encounter a problem with an application or a script that is not clear straight away. After the normal troubleshooting, it can be helpful to see the actual system calls that occur when that script executes. Using a tool like strace (manpage) can help you in identifying what is causing the system to slow down or misbehave.

In this blogpost I'll show you some examples where strace can be useful for you. Most of it will be with PHP code but they're easy enough anyone can understand them.

Installing strace

Strace isn't installed by default on most distributions. To install, do a simple yum install strace when on CentOS/Red Hat or apt-get install strace on Debian/Ubuntu systems.

Getting output from strace

You can use strace in two different ways. You can attach it to an already running process or you can use it to start a custom application or script and follow all system calls from the very beginning.

In short, here's how it goes. If you want to start your application and troubleshoot it from beginning to end, you do this:

~# strace -f $command
~# strace -f php -q somefile.php

Or you want to attach to a running process, use this:

~# strace -f -p $pid
~# strace -f -p 8151

The -f parameter tells strace to follow any children or processes that are spawned/forked from the application.

Standard usage of strace

By default, strace will show you -all- system calls that your application or script is performing. That can get pretty overwhelming, but it's a good place to start. Take for instance the following simple script called 'test1.php'.

<?php
   /* Simple buggy script */
   for ($i = 0; $i < 5; $i++)
      sleep(1);
?>

The command above will simply iterate over the loop and sleep (do absolutely nothing) for 1 second on each iteration. So, this would do nothing for exactly 5 seconds. Best script I've ever written! ;-)

If you were to strace the execution of the script, it would look like this.

~# strace php -q test1.php
execve("/usr/bin/php", ["php", "-q", "test1.php"], [/* 23 vars */]) = 0
brk(0) = 0xa036000
access("/etc/ld.so.preload", R_OK) = -1 ENOENT (No such file or directory)
open("/etc/ld.so.cache", O_RDONLY) = 3
...
nanosleep({1, 0}, {1, 0}) = 0
...
nanosleep({1, 0}, {1, 0}) = 0
...
nanosleep({1, 0}, {1, 0}) = 0
...

At first, it's overwhelming. You don't need to understand every system call (sure, it helps, but the names of most functions explain the actions anyhow), but the execution can show you a few remarkable things straight away. When it's running, you'll see the application hang every second with the nanosleep({1, 0}, {1, 0}) = 0 message. The nanosleep() is the system call for the sleep() method in PHP. Without having seen the source code of the application, this tells you the application specifically requested to sleep() there and waited.

But since the output was a bit too much to work with, let's look at filtering it.

Filtering the strace output to something more meaningfull

Have a look at the following simple script as a new example.

<?php
   $fp = fsockopen('www.google.be', 80, $errno, $errstr, 30);
?>

It opens a socket to www.google.be over port 80, a very simple action. The same kind of action would occur when connecting to remote MySQL databases, accessing a remote API via curl/fsockopen/file_get_contents/... If you strace it, the network connectivity won't immediately be obvious because of all the other sysetm calls. That's why you can pass some extra arguments to strace to make that more clear.

~# strace -e trace=network php -q test2.php
...
connect(3, {sa_family=AF_INET, sin_port=htons(53), sin_addr=inet_addr("193.239.211.254″)}, 28) = 0
send(3, "<b\1\0\0\1\0\0\0\0\0\0\3www\6google\2be\0\0\1\0\1″, 31, MSG_NOSIGNAL) = 31
recvfrom(3, "<b\201\200\0\1\0\2\0\4\0\4\3www\6google\2be\0\0\1\0\1\300″..., 1024, 0, {sa_family=AF_INET, sin_port=htons(53), sin_addr=inet_addr("193.239.211.254″)}, [16]) = 219
socket(PF_INET, SOCK_STREAM, IPPROTO_IP) = 3
connect(3, {sa_family=AF_INET, sin_port=htons(80), sin_addr=inet_addr("74.125.79.94″)}, 16) = -1 EINPROGRESS (Operation now in progress)
getsockopt(3, SOL_SOCKET, SO_ERROR, [0], [4]) = 0

This shows you a few specific things. The first connect, send and recvfrom are to translate the hostname "www.google.be" to the IP address "74.125.79.94″, so DNS resolving. The last connect is the actual connection attempt based on the IP address to the Google systems. It shows you the IP, the port and the state (Operation now in progress).

If I were to change my script to connect to port 85 instead of 80, the results would be quite different, since the connection to port 85 will not succeed.

~# strace -r -e trace=network php -q test2.php
...
0.000107 connect(3, {sa_family=AF_INET, sin_port=htons(85), sin_addr=inet_addr("74.125.79.94″)}, 16) = -1 EINPROGRESS
20.999554 getsockopt(3, SOL_SOCKET, SO_ERROR, [110], [4]) = 0

I've changed the strace call a bit, by adding the '-r' parameter. That shows you timestamps of how long that specific call took relative to the previous call. By stracing, you will notice immediately that your application or script is waiting for the connect() call to finish. That may not immediately be obvious from just executing the script via php, but it does show clearly when using strace. In this case, it took exactly 20s for that connection to timeout. Either that's because your connection is blocked by a firewall, the service on the other end did not respond or took too long to respond.

Other means of filtering the output are:

• -e trace=file: show all system calls that take a file as argument. This can show you where a lot of disk I/O may occur. Every open, stat, chmod, unlink, ... shows up there;
• -e trace=process: track forks of the script
• -e trace=open: show all calls that invoke the open() method.

You can substitute the -e trace=xxx with any kind of system call. That can be open, close, read, write, stat, chmod, unlink, ...

Spotting query output in strace

When using strace to trace an Apache process for instance, you can often find output like this.

munmap(0xb7973000, 1314816) = 0
brk(0x1027b000) = 0x1027b000
poll([{fd=740, events=POLLIN|POLLPRI}], 1, 0) = 0 (Timeout)
write(740, "\200\0\0\0\3SELECT `something` "..., 132) = 132
read(740, "\1\0\0\1!n\0\0\2\3def\fdbname\35something"..., 16384) = 3931
mmap2(NULL, 1314816, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0xb7973000
...
munmap(0xb77f0000, 1314816) = 0
poll([{fd=740, events=POLLIN|POLLPRI}], 1, 0) = 0 (Timeout)
write(740, "\200\0\0\0\3SELECT `something` "..., 132) = 132
read(740, "\1\0\0\1!n\0\0\2\3def\fdbname\35something"..., 16384) = 3931
mmap2(NULL, 1314816, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0xb77f0000

If you look at (parts of) the strace above, you will notice the lines in bold. In this case, a script kept on looping and launching a query to the backend database. A loop like that only shows up as very high CPU usage of an Apache process, with strace you can gain the insights in to why that process is looping or causing a high load. Again, you don't need to understand every call that strace will output, but you will notice the SQL statements that are sent in plain text over the wire.

Why use strace?

I often fall back to strace to troubleshoot applications that are running for an abnormally long time (using strace -p PID, where PID is the process ID of that particular script or application). It's very useful to be able to attach to a running process and see what it is doing. Even when you don't fully understand the output, you do notice repetitive tasks very easily, which could indicate the program is in some kind of loopt.

In hosting environments, it's not uncommon for me to strace certain Apache or FastCGI processes to detect why it's causing an abnormally high load or why it's running for an abnormally long time.

When you're debugging applications that appear to stop or crash halfway with no immediate error message (because errors are being suppressed somewhere in the middle of application execution), stracing it could be a good way to see the final system calls before the application halts.

If you have any other useful strace-tips, please share them in the comments!

I've been cursing at this quite often. Since this is a tech-related blog with code snippets, command line arguments, ... running it on wordpress means that it always results in having characters replaced by the WordPress default formatting. In the end, that means that a single quote ' or a double quote " would always get replaced by a better-looking HTML character. That made copy/pasting from this blog to a terminal near impossible.

Previously, I would just hack the WordPress core (for WordPress 3.3 and 3.2 or 3.1) by editting some files. But any update would of course undo my change, since hacking the core of anything makes kittens cry.

Therefore it was time to create a solid fix, one that would last through the different updates. So I present to you, my Undo WordPress Default Formatting plugin for WordPress!

It's a very simple thing but I'm sure it applies to a lot of techblogs out there running on WordPress.

Download directly: undo_wordpress_default_formatting.tar.gz
Plugin page at WordPress: http://wordpress.org/extend/plugins/undo-wordpress-default-formatting/

I'll keep the subversion repository up-to-date, so feel free to report any findings via the page at wordpress.org.

Ivan Torres Fally

Ce jeudi 2 février 2012 à 19h se déroulera la deuxième séance bruxelloises des Jeudis du Libre de Belgique.

• Le sujet de cette séance : l’intégration continue avec Jenkins (Hudson)
• Thématique : développement
• Public : développeurs, intégrateurs, agilitateurs
• L’animateur conférencier : Ivan Torres Fally (Site perso)
• Lieu de cette séance : Betagroup Coworking Brussels at ICAB Business & Technology Incubator (cf. ce plan sur le site d’Openstreetmap).

 

La participation sera gratuite et la séance sera suivie d’un restau dans le quartier (merci de préciser si vous restez manger lors de l’inscription) si affinités.

Si vous êtes intéressé(e) par ce cycle mensuel, n’hésitez pas à consulter l’agenda et à vous inscrire sur la liste de diffusion afin de recevoir systématiquement les annonces, ou à nous contacter à l’adresse jeudisdulibre@loligrub.be

Pour rappel, les Jeudis du Libre se veulent des rencontres autour de thématiques des Logiciels Libres. Les rencontres bruxelloises se déroulent chaque premier jeudi du mois, et sont organisées avec le concours de l’A.S.B.L. LoLiGrUB !

Description

Jenkins (Hudson) est un outil d’intégration continue permettant entre autres d’améliorer la qualité d’un logiciel en accélérant et en automatisant aussi bien les processus de contrôle du code que le déploiement dans les environnements de test, de recette et pourquoi pas de production.

La présentation sera l’occasion de proposer une interprétation des concepts de base de l’intégration continue mais surtout un retour d’expérience sur son utilisation dans un cadre professionnel avec comme cas d’utilisation :

• le contrôle de qualité du développement d’un S.I. en Java
• l’automatisation du packaging d’une application smartphone multi-plateforme
• la présentation de quelques plug-ins intéressants
• l’intégration avec des outils familiers tels que Maven, Ant ainsi que des scripts Bash

L’objectif de la présentation sera dans un premier temps de vous familiariser avec les concepts et les idées derrière l’intégration continue et ensuite de vous donner un feedback permettant de vous faire un avis objectif sur l’intérêt de Jenkins (Hudson).

Les démonstrations porteront donc essentiellement sur des situations réelles bien précises en entreprise mais tenteront d’aller plus loin dans les limites des connaissances du conférencier.

 

Billetterie en ligne pour Jeudis du Libre de Bruxelles: l’intégration continue avec Jenkins (Hudson) produit par Eventbrite

Lieu

BetaGroup Coworking Brussels

ICAB Business & Technology Incubator

Betagroup Coworking Brussels
ICAB Business & Technology Incubator
Witte Patersstraat 4 rue de Pères Blancs1040 Brussel-Bruxelles (Etterbeek)
Arrêt Arsenal Tram 7, 25; Bus 34.

L’animateur conférencier

Ivan Torres Fally travaille dans une société de services financiers en tant que développeur.

This year the ULB has kindly offered us the use of an additional new building on their campus. It's the K building and it's located on the other side of the parking from the Jason Lecture hall.

read more

When we designed a new governance structure for the Drupal Association last year, we decided that most of the board is selected through a nominating committee with the goal to carefully balance many factors like needed skills and geographical and sector representation. However, it was also deemed important that we have directors chosen directly by the Drupal community to make sure that the community is always well-represented.

I'm excited that we're holding our first open community elections. Two community "at large" directors will be elected to the Drupal Association Board of Directors. If you'd like to consider running, please have a look at the "At-large" nominations page. And if you're a Drupal community member, please make time to participate in discussions with candidates and of course to vote, starting January 26. (This process was vetted openly in the community by the Elections Committee and numerous community volunteers at http://groups.drupal.org/drupal-association.)

Your participation will help us take this next important step in implementing a new improved governance structure to strengthen the Drupal Association. Thanks!

The schedule for the Mono devroom at FOSDEM 2012 has been finalized:

• (11:00 – 11:45) Enough Debian packaging knowledge to hurt yourselves slightly less than you do already (Jo Shields)
• (12:00 – 13:00) Mono – State of the Union (Miguel De Icaza)
• (13:00 – 14:00) *** Lunch break ***
• (14:00 – 14:30) Gluon (Federico Di Gregorio)
• (14:40 – 15:25) Banshee: Past, Present, Future and the Crazy stuff (Bertrand Lorentz & Olivier Dufour)
• (15:40 – 16:25) IronPython: Bringing the dynamic world to the CLR (Carlos Alberto Cortez Guevara)
• (16:40 – 17:30) MonoGame (Dominique Louis, Dean Ellis & Kenneth Pouncy)
• (17:45 – 18:15) XWT (Lluis Sanchez)
• (18:30 – 19:00) MonoMac (Miguel De Icaza)

Full details (with talk abstracts) should be available on the FOSDEM website. As always, if you want to see a talk, show up early, once the room is full, you’re out of luck!

Date to put in your agenda: Feb 4, 2012.

Dear all,

I am going to do a presentation about “Red Hat Global Support Services” in Brussels in January 2012.
If your company or you are based in BeNeLux and are interested by this presentation, just let me know and I will try to arrange a Meeting for you.

Ref  : http://www.redhat.com/support

Ref : https://access.redhat.com

BR

Frederic

Traduction française

Dear copyright industry, I'm a pirate. I'm the typical user you are fighting. I'm downloading everything and not giving you one single penny. I don't even attend concert. You hate me and it's reciprocal.

When I discovered high-speed Internet, I was a naive young innocent. I was downloading to discover new stuffs. Whenever I liked something, I would go to the shop and buy the CD. I discovered lot of music thanks to the pirate networks. Randomly or following advices. In the end, I bought something like 200 CDs. The first group I've joined on Audioscrobbler was called "I still buy CDs". But today I regret that. I'm asking everyone to not buy CD any more. Not a single one !

Because you are not offering a good service

When I want to discover an artist or a movie, I'm heading to The Pirate Bay, I launch a search and I click. In less than 10 minutes, I've a full movie on my disk. In 20, I've the complete discography of an artist.

I would pay for such a service if it is as simple, as fast and, unlike the Bay, if it can make some guarantees on the quality. But you don't offer that. Instead, you are trying to build fences and limitations. You are asking for huge amount of money only through credit card and you don't have half the music I'm looking for. That's not convenient and it's more expensive.

I don't even talk about CDs any more. This is now only a huge quantity of plastic waste, sitting in my living room. They are expensive, they become unreadable through the years or, thanks to DRM, they are unreadable since the first day.

In summary, you are offering less for a more expensive price.

Because you don't use my money well

I've probably spent something like 2000€ for my CDs. You need to add the taxes on all the blank CDs I used to burn Linux iso. From that money, how much went to the artists and their studio? 100€? 200€? Everything else was probably diluted in stuffs I don't need: packaging, distribution, transport, marketing, …

Your companies are in the top richest ones. The artists that are the most downloaded live in huge luxury houses. Others are dead. Don't you find it a bit shameful to try the "bad pirates are killing the poor artists" story?

I'm sorry but I don't think you need my money. I've showed my support to small artists with Flattr, Jamendo, CDbaby or Magnatune. For everything else, you will have to live without my wallet.

Because you are messing with my life

That's it. Every penny I will give you will be used against me. Firstly, by making it difficult for me to use what I buy. Zoned DVDs, encrypted movies on the DVD requiring illegal software to be read under Linux or DRM to be sure I'm not able to listen to a CD.

Worst, you will use my money to sue me in court because I would have downloaded something that I didn't want to buy anyway! With the change left, you will pay lobbyists to ensure the governments make stupid and dangerous laws.

Do you want me to pay lawyers to sue myself and lobbyists to make laws to send me in jails? Really?

Because you are destroying the whole society

Messing with my life was not enough. You are even trying to destroy one of the pillar of our society: education. Your heavy marketing is starting to work, people now understand the importance of "intellectual property" and that "sharing is bad".

Bloody ignorant morons.

Thanks to you, schools are now afraid to give lectures in case there are some copyrighted materials in them. Teachers fear to be sued. To the point where giving the strictly minimal lesson is better than giving some examples. Famous works are not part of the education any more.

Some teachers themselves start to consider their lectures as "copyrighted material", refusing to share it with colleagues. And when they attend training sessions, offered by the state and paid with public funds, it is to hear that the material of the session can be read but has to be bought if the teachers want to use them in their own classroom.

You are destroying the very symbol of civilisation: the enjoyment of knowledge, the joy of sharing, the cooperation and the education. I will never forgive you for that. Never. If I'm not taking action right now, my children will be more afraid of reading a copyrighted book than stealing in a shop or hitting someone with a knife. Those crimes are anyway less punished by the law than sharing a song on the internet.

How can you look at yourself in the mirror after that? How can you still have a peaceful sleep ?

Because your time has come

If I'm a pirate, it's not to have some cheap music. It is because the time has come for you to fuck off. In your arrogance, you are hurting the fundamental value of freedom only to save your little petty interests.

The only comfort is to know that you will disappear soon. And nobody will miss you.

Pirately yours,


Picture by arbyreed
Traduction française

English translation

Chers industriels de la musique, du cinéma et du divertissement, je suis un pirate. Je suis l'archétype de l'internaute que vous combattez. Je télécharge tout et ne vous reverse pas un seul centime. Je ne vais même pas aux concerts. Vous me détestez et c'est réciproque.

Lorsque j'ai découvert l'Internet haut-débit, j'étais innocent, naïf. Je téléchargeais pour découvrir et, si j'aimais bien, j'achetais le CD. J'ai découvert énormément de musique via les réseaux pirates. Au hasard des mots-clés ou en suivant des recommandations. Au total, j'ai acheté légalement près de 200 CDs. Le premier groupe que j'ai rejoint sur Audioscrobbler s'intitulait « I still buy CDs ». Mais je le regrette. Et je demande à tout le monde de ne plus acheter de CD. Plus un seul.

Car vous ne proposez pas un service pertinent

Aujourd'hui, lorsque je veux découvrir un artiste ou un film, je vais sur The Pirate Bay, je lance une recherche et je clique. En moins de 10 minutes, j'ai un film entier. En 20 j'ai la discographie complète d'un artiste.

Je serais prêt à payer pour un tel service s'il est aussi simple, aussi rapide et, contrairement à la Baie, s'il me garantit la qualité de ce que j'écoute. Mais vous n'offrez pas cela. Vous limitez, vous demandez des sommes folles payables uniquement par carte de crédit. Ce n'est pas pratique et c'est plus cher.

Je ne parle même plus des CDs, qui encombrent mon salon, qui sont un fardeau à chaque déménagement, qui coûtent un prix démesuré, qui se dégradent avec le temps et qui, dernièrement, sont bardés de verrous numériques rendant le tout inutilisable.

Bref, vous offrez un service plus cher et moins bien que l'existant.

Car vous ne redistribuez pas mes sous correctement

Mes 200 CDs m'ont probablement coûté un total de 2000€. Auxquels il faut ajouter les taxes sur les CDs vierges que j'achetais pour installer Linux. Sur cette somme, combien ont servi à rémunérer les artistes et l'enregistrement ? 100€ ? 200€ ? Le reste s'est certainement dilué dans des postes dont je me passe très bien aujourd'hui : packaging, distribution, transport, marketing, …

Vos sociétés sont parmi les plus riches du monde. Les artistes que la majorité d'entre nous écoutent habitent dans des gigantesques villas. Les autres sont morts. Vous osez malgré tout me faire le larmoyant numéro du méchant internaute qui tue les artistes crêve-misère ?

Désolé, je pense que vous n'avez pas besoin de mon argent. J'ai volontiers soutenu les petits artistes via Flattr, Jamendo, CDbaby, Magnatune voire même directement. Pour le reste, vous vous passerez très bien de mes sous.

Car vous me pourrissez la vie

Si je vous donne mes sous, vous allez les utiliser à me pourrir la vie. Tout d'abord en rendant l'utilisation de mes achats difficiles voire impossible. Les DVDs ne fonctionnent que dans certains pays. Sous Linux, je dois également installer un logiciel illégal pour pouvoir les décrypter et je ne parle pas des DRM. Bref, quoique je fasse, je suis dans l'illégalité.

Pire, vous utiliserez mes sous en m'attaquant en justice parce que j'ai téléchargé une musique que je n'aurais de toutes façons pas achetée. Puis, vous dépenserez la monnaie restante en lobbying auprès des gouvernements pour faire passer des lois stupides voire liberticides et dangereuses.

Vous voudriez que je paie les avocats qui vont me trainer en justice et les lobbyistes qui vont écrire les lois pour me jeter en prison ? Et puis quoi encore ?

Car vous êtes en train de détruire la société

Mais il ne vous suffisait pas de me pourrir la vie. Il a fallu que vous tentiez de détruire le fondement même de notre société, le système éducatif. Votre matraquage et lobbying incessant est un succès, la majorité de la population a bien compris l'importance de la « propriété intellectuelle ». Copier, c'est mal !

Bande de sombres crétins ignorants.

Grâce à vous, les écoles n'osent plus donner cours en utilisant du matériel élaboré. Les enseignants ont une peur bleue de se faire attaquer en justice. À tel point que donner un mauvais cours est préférable que prendre le risque d'utiliser une œuvre copyrightée.

Certains professeurs eux-mêmes ne partagent plus leurs cours avec les nouveaux ou les stagiaires, arguant que le travail est « leur propriété intellectuelle ». Et lorsque les enseignants suivent des formations dispensées par l'état, financée par les deniers publics, on annonce aux participants que le matériel de la formation peut être consulté gratuitement mais doit être acheté pour pouvoir être utiliser en classe.

Vous être en train de détruire ce que la civilisation a de plus cher: le plaisir de la culture, le partage de la connaissance, l'entraide, le développement personnel et l'éducation. Cela, je ne le vous pardonnerai jamais. Si je n'agis pas, mes enfants auront plus peur de copier un livre copyrighté que de voler dans un étalage ou de donner un coup de couteau. Crimes qui sont d'ailleurs moins punis par la loi que le partage de musique sur Internet.

Pouvez-vous encore vous regarder dans une glace après ça ? Avez-vous le sommeil tranquille ?

Car votre heure est venue.

Si je suis un pirate, ce n'est pas pour avoir de la musique à moindre coût. C'est parce que votre temps est venu de disparaitre. Parce que votre arrogance et votre suffisance n'ont d'égales que votre volonté de faire du mal à la société pour défendre vos misérables petits intérêts personnels.

La seule chose qui me console c'est de savoir que vos heures sont comptées. Et que personne ne vous regrettera.

Piratement vôtre,


Photo par arbyreed
English translation

(Source: pastebin.com)

For those who (still) don’t know pastebin.com, it’s  a website mainly for developers. Its purpose is very simple: You can “paste” text on the website to share it with other developers, friends, etc. You paste it, optionally define an expiration date, if it’s public or private data and your are good. But for a while, this on-line service is more and more used to post “sensitive” information like passwords or emails lists. By “sensitive“, I mean “stolen” or “leaked” data. Indeed, pastebin.com allows anybody to use their services without any authentication, it’s easy to remain completely anonymous (if you submit data via proxy chains, Tor or any other tool which takes care of your privacy)

In big organizations, marketing departments or agencies learned how to use social networks for a long time. They can follow what has been said about their products and marketing campaigns. In my opinion, it is equally important to follow what’s posted about your organization on pastebin.com! Many people are looking for interesting data on pastebin.com from an offensive point of view. Let’s see how this can also benefit to the defensive side.

For me, pastebin.com became an important source of information and I keep an eye on it every day. But, due to the huge amount of information posted every minute, it is impossible to process it manually. Of course, you can search for some keywords but it’s totally inefficient. In a first time, I grabbed and processed some HTML content using the classic UNIX tools. Later, I found a nice Python script developed by Xavier Garcia: python.py. It checks continuously for data leaks on pastebin.com using regular expressions. I kept it running for a while on a Linux box and it did a quite good job but I needed more! Xavier’s script send the found “pasties” on the console. It is possible to dump the detected pasties by sending a signal to the process. Not always easy. That’s why I decided to go a step further and write my own script! The principle remains the same as the script in Python (why re-invent the wheel?) but I added two features that I found interesting:

• It must run as a daemon (fully detached from the console) and started at boot time.
• It must write its finding in a log file.

The next step sounds logical: If you have a log file, why not process it automatically: Let’s monitor pastebin.com within your SIEM! If you find information posted on pastebin.com, it could be very interesting to be notified (a great added-value for your DLP processes). My script generates Syslog messages and (optionally) CEF (“Common Event Format“) events which can be processed directly by an ArcSight infrastructure. Syslog messages can be processed by any SIEM or log management solution like OSSEC (see below). It is now possible to completely automate the process of detecting potentially sensitive leaked data and to generate alerts on specific conditions.

First install the script on a Linux machine. Requirements are light: a Perl interpreter with a few modules are required (normally all of them are already installed on recent distribution) and a web connectivity to http://pastebin.com:80. If you are behind a proxy, you can define the following environment variable, it will be used by the script:

  # export HTTP_PROXY=http://proxy.company.com:8080

The script can be started with some useful options:

  Usage: ./pastemon.pl --regex=filepath [--facility=daemon ] [--ignore-case][--debug] [--help]
                       [--cef-destination=fqdn|ip] [--cef-port=<1-65535>] [--cef-severity=<1-10>]
  Where:
  --cef-destination : Send CEF events to the specified destination (ArcSight)
  --cef-port        : UDP port used by the CEF receiver (default: 514)
  --cef-severity    : Generate CEF events with the very easy to process and can be specified priority
                      (default: 3)
  --debug           : Enable debug mode (verbose - do not detach)
  --facility        : Syslog facility to send events to (default: daemon)
  --help            : What you're reading now.
  --ignore-case     : Perform case insensitive search
  --regex           : Configuration file with regular expressions (send SIGUSR1 to reload)

Once running, the script scans for newly uploaded pasties and search for interesting content using regular expressions. There is no limitation on the number of regular expressions (defined in a text file). To not disturb pastebin.com webmasters, the script waits a random number of seconds between each GET requests (between 1 and 5 seconds). There is only one mandatory parameter ‘–regex‘ which gives the text files with all the regular expressions to use (one per line). If one of the regular expressions matches, the following information will be sent to the local Syslog daemon:

  Jan 16 14:43:24 lab1 pastemon.pl[29947]: Sending CEF events to 127.0.0.1:514 (severity 10)
  Jan 16 14:43:24 lab1 pastemon.pl[29947]: Loaded 17 regular expressions from /data/src/pastemon/pastemon.conf
  Jan 16 14:43:24 lab1 pastemon.pl[29947]: Running with PID 29948
  <time flies>
  Jan 16 15:57:48 lab1 pastemon.pl[29948]: Found in http://pastebin.com/raw.php?i=hXYg93Qy : CREATE TABLE (9 times) -- phpMyAdmin SQL Dump (1 times)

All matching regular expressions are listed with their number of occurrences. This can be easily processed by OSSEC using the following decoder:

  <decoder name="pastemon">
    <program_name>^pastemon.pl</program_name>
  </decoder>

  <decoder name="pastemon-alert">
    <parent>pastemon</parent>
    <regex>Found in http://pastebin.com/raw.php?i=\.+ : (\.+) \(</regex>
    <order>data</order>
  </decoder>

The first regular expression is stored in the OSSEC “data” variable to be used as  conditions in rules. Here is an example: The rule #100203 will trigger an alert if some yahoo.com email addresses are leaked in pastebin.com. (Note: This regular expression must be defined in the script configuration file!)

  <rule id="100203" level="0">
    <decoded_as>pastemon</decoded_as>
    <description>Data found on pastebin.com.</description>
  </rule>

  <rule id="100204" level="7">
    <if_sid>100203</if_sid>
    <description>Detected yahoo.com email addresses on pastebin.com!</description>
    <extra_data>@yahoo\.com$</extra_data>
  </rule>

If you have an ArcSight infrastructure, you can enable the CEF events support. The same event as above will be sent to the configured CEF destination and port:

<29>Jan 16 15:57:48 CEF:0|blog.rootshell.be|pastemon.pl|v1.0|regex-match|One or more regex matched|10|request=http://pastebin.com/raw.php?i=hXYg93Qy destinationDnsDomain=pastebin.com msg=Interesting data has been found on pastebin.com.
cs0=CREATE TABLE cs0Label=Regex0Name cn0=9 cn0Label=Regex0Count cs1=-- phpMyAdmin SQL Dump cs1Label=Regex1Name cn1=1 cn1Label=Regex1Count

To process the CEF events on ArcSight’s side, configure a new SmartConnector, a new UDP CEF receiver and the events should be correctly parsed:

(Click to enlarge)

That looks great! But the next question is: “What to look for on pastebin.com?“. Well, it depends on you… Based on your organization or business, there are things that you can’t miss. Here is a list of useful regular expressions that I often use:

RegEx                                                                  Purpose
---------------------------------------------------------------------  -----------------------------------
company\.com                                                           Your company domain name
@company\.com                                                          Corporate e-mail addresses
CompanyName                                                            Company name
MyFirstName MyLastName                                                 Your full name
@xme                                                                   Twitter account
192.168.[1-3].[0-255]                                                  IP addresses ranges
anonbelgium                                                            Hackers groups
#lulz                                                                  Trending Twitter hashtags
#anonymous
#antisec
-----BEGIN RSA PRIVATE KEY-----                                        Interesting data!
-----BEGIN DSA PRIVATE KEY-----
-----BEGIN CERTIFICATE-----
-- MySQL dump                                                          Interesting dumps!
belgium                                                                My country
city                                                                   My city
((4\d{3})|(5[1-5]\d{2})|(6011))-?\d{4}-?\d{4}-?\d{4}|3[4,7]\d{13}      Credit cards

If you have interesting regular expressions or ideas, feel free to share!

Source is available here. As usual, this is provided “as is” without any warranty. Happy monitoring!

Traduction en français disponible

Lot of people think that we are on the verge of a revolution, that the foundation of the society will be shaken. I don't think so.

For me, the revolution has already happened, we are at the end of a transition period. We need time to realize it but the changes are there, unavoidable.

The revolution I'm speaking about is well known under the name "industrial revolution". Started in the middle of the XIXth century, it became prominent with the launch of the Ford T in 1908. And ends today, with Internet and the worldwide network.

For ages, humans have worked. That work was, granting a few exceptions, mainly proportional to the final result. As a farmer cultivated more field, he had a better harvest. A craftsman was working more hour to produce more. As the money^[1] earned was directly proportional to the result, we could say that the more we work, the more we earn. One hour of work was roughly equivalent to a given sum of money.

The industrial revolution changes completely this paradigm. The price of a good becomes inversely proportional to the number of produced goods. If Henri Ford had produced only one Ford T, it would worth millions. But the more he was producing, the better the return was. To the point where buying a brand new good is often cheaper than repairing and existing one.

The industrial world is thus characterized by the desire to duplicate as much as possible in order to lower the unit price. This world is not directed by the direct production but by speculation. To launch a business, you need a complex estimation about when you will have a return on your initial investment. The more you produce, the richer you are. The richer you are, the more you can speculate and thus become even richer. Money attracts money, the society itself is directed toward speculation.

The funny side is that the first beneficiaries of this revolution don't want others to share the pie and try to convince everyone that the old rules still apply. Work is still paid by the hour, even though the principle itself is not founded any more and leads to hidden conflicts of interest.

It is also noteworthy that, even though the industrial revolution funding principle is to duplicate as much as you can, tools are created to avoid that very same duplication: Patents, intellectual property and, later, DRM. Depending on the context, duplication will be translated by production, growth or by counterfeiting, piracy.

Internet, digital products, globalisation and 3D printers are only the logical consequence of this duplication revolution. Duplication tools are cheap, easily accessible. Everyone can become a producer, everybody can benefit from the revolution and that's why we can say it has succeeded. Dear big manufacturers, being first granted you a lot of benefits during 150 years. Now please be kind enough to not fight against the unavoidable spread of your privileges.

That's it, we are entering the post-industrial era. I've no idea what it willlooks like. I will probably never know if we are currently living a new transition period of the founding of a big, Millennial Worldwide Society.

What I know is that the industrial era is coming to a end, that we need to redefine fundamentals like money, work, wealth, property, power. That those who took them for granted will not be happy. But who cares about them anyway ?


Pictures by ialla and danmachold
Traduction en français disponible

English translation available

Beaucoup pensent que nous sommes à la veille d'une révolution, que des bouleversements vont secouer notre société. Je ne le pense pas.

À mes yeux, la révolution est terminée, nous sommes à la fin de la période de transition. Il faut juste un peu de temps pour nous en rendre compte mais le changement est déjà présent, irrémédiable.

Cette révolution est bien connue des historiens et porte le nom de « révolution industrielle ». Commencée aux alentours du XIXème siècle, elle prend un virage décisif avec la production en série de la Ford T en 1908. Et s'achève aujourd'hui, avec Internet et le réseau mondial.

Pendant des millénaires, les hommes ont effectué un travail. Ce travail était, à quelques exceptions près, proportionnel au résultat. Plus un paysan labourait une grande superficie de champs, plus grande était sa récolte. Plus un artisan travaillait, plus grande était sa production. Comme l'argent^[1] gagné était généralement proportionnel au résultat, on pouvait sans trop se tromper affirmer: plus on travaille d'heures, plus on gagne de l'argent. Une heure de travail est équivalente à une somme d'argent.

La révolution industrielle bouleverse complètement la donne. Le prix d'un bien quelconque devient inversement proportionnel au nombre de biens produits. Si Henri Ford avait produit une seule Ford T, elle vaudrait des millions. Mais plus il en produisait, plus il rentabilisait son infrastructure sans effectuer de travail supplémentaire. C'est vrai à tel point que réparer un produit abimé coûte de nos jours plus cher qu'acheter le même produit neuf.

Le monde industriel se caractérise donc par la volonté de reproduire autant que possible un bien de consommation afin d'en faire baisser le coût unitaire. Ce monde n'est donc plus dirigé par la production directe mais bien par la spéculation. À chaque fois qu'un entrepreneur lance une idée, il doit faire de savants calculs pour estimer en combien de temps il rentrera dans ses frais. Plus on produit, plus on est riche, plus on a le loisir de spéculer et plus on peut devenir encore plus riche. L'argent appelle l'argent, la société est entièrement tournée vers la spéculation.

Ce qui est amusant c'est que les premiers bénéficiaires de cette révolution industrielle se voient eux-mêmes comme des exceptions et essaient, de toutes leur force de maintenir la majorité de la population dans la croyance que l'ancien système est d'application. Le travail est toujours payé « à l'heure » alors même que ce principe n'a plus aucune réalité tangible et induit des conflits d'intérêts latents.

Il est également remarquable que, depuis le début de la révolution industrielle dont le principe est de reproduire, des outils sont mis en place afin d'empêcher cette même reproduction par des concurrents: brevets, principe de la propriété intellectuelle et plus tard DRM. Dans la bouche des industriels, reproduction devient production, croissance ou contre-façon, piratage, selon le contexte.

Internet, la numérisation des produits, la globalisation et les imprimantes 3D ne sont que l'aboutissement logique de cette révolution de la reproduction. Les outils de reproduction sont accessibles à tous, chacun peut devenir son propre industriel. La révolution s'achève car elle touche enfin tous les citoyens. Messieurs les industriels, vous avez bénéficié grandement de cette transition en étant les premiers. Ayez l'élégance de ne pas vous accrocher, de ne pas refuser aux autres ce dont vous avez profité pendant 150 ans.

Voilà, nous entrons doucement dans la période post-industrielle. Ce que nous réserve cette période, je n'en sais rien. Pas plus que je ne saurai jamais si elle sera une courte période de transition de 200 ans ou si elle va jeter les fondements d'une société mondiale millénaire.

Tout ce que je sais ce que le monde industriel s'achève, qu'il nous faut redéfinir des notions aussi fondamentales que l'argent, le travail, la richesse, la propriété, le pouvoir, que ceux qui les considéraient comme acquis voient cela d'un mauvais œil. Mais leur avis est-il tellement important ?


Images de ialla et danmachold
English translation available

Puppet modules…  How I feel about them in a dot file:

digraph PuppetModules {
  node [
    fontname = "Bitstream Vera Sans"
    fontsize = 10
    shape    = "record"
  ]
  edge [
    fontname = "Bitstream Vera Sans"
    fontsize = 10
  ]
  question [label="Do I need to edit a file in your module for changing settings?", shape="oval"]
  ok  [label="Great.", shape="oval"]
  bah [label="You are doing it WRONG!", shape="oval"]

  question -> ok [label="No"]
  question -> bah [label="Yes"]
}

Read on if you want a rendered version.

This year, Load will host LPI exams again.
LPI offers level 1, level 2 and level 3 certification exams during the Loadays event
Exams will be in English and include the following certifications:

Exams

read more

kexec's mechanism is most commonly used with kdump. Basically with kdump, when a crash or panic occurs, a new kernel is booted after the crash while the memory is preserved from the previous kernel's runtime. The new kernel can then capture this data and generate the dump which then can go to local disk, remote disk or anywhere else for that matter. In order to use kdump, you basically have to allocate/reserve memory for this dump kernel. This is done by adding crashkernel=xxx@yyy to the grub command line when booting. The crash kernel image is then loaded and will be executed when a crash or panic occurs. Even though kdump is a bit cumbersome to set up, it allows for really great flexibility and is very powerful in helping with debugging issues.

For those interested in kdump, there's a good blog out there test kdump on Oracle Linux.
Or for those that just want to read the documentation that's part of the Linux kernel tree : kdump.

Anyway, this entry is not about kdump. kdump is great but I wanted to talk about the use of kexec proper and how it can help with doing fast reboots of your systems. Both Oracle Linux 5 and Oracle Linux 6 have support for reboot to use kexec as the reboot mechanism (see /etc/init.d/halt for details). When a standard reboot command is executed, init goes to 6 and /etc/init.d/halt gets run. This script, when it sees that kexec has been configured with a kernel image, will just execute kexec -e. In a standard reboot (not reboot -f) the normal shutdown scripts get executed and at the end where the system normally does a reset.
This reset then makes the system hard reset, jump into the bios, does a memory test, finds devices, initialize the devices and firmware, boot the bootdevice bootloader, start the kernel.

To set up kexec you should run the following command shortly after you boot the system. If you want to automate this, it makes sense to add this to your rc scripts. We will look at integrating this more into the OS management scripts for Oracle Linux to make it easier for the system administrators.

kexec -l --append="`cat /proc/cmdline`" 
    --initrd=/boot/initrd-`uname -r`.img /boot/vmlinuz-`uname -r`

Once this is done, the new kernel image is prepared, memory is allocated and you now can do one of 2 things :

- run reboot : halt at the tail end of a normal reboot (shutdown all services) will execute directly into this new kernel image, exactly the same way as you booted the OS to get to this point.

- you wish to do a very fast reboot without shutdown (reboot -f). then you do

sync; umount -a ; kexec -e

The total amount of time saved is highly dependent on your server. Basically time a system startup all the way to grub executing the kernel image, that's the amount of time you will save on a subsequent reboot. This can range from a number of seconds (15,20) to, sometimes, several minutes.

One caveat with the use of kexec and instant restarts without going through device resets is that, in some cases, the devices might act badly or the driver might not be doing the right thing. Before you really use this on your system, test it first to ensure that the drivers for the hardware you have and the devices themselves are doing the right thing (tm).

You can find more info about kexec in this article written a number of years ago : kexec article.

I am planning on writing a entry next about how to use this with Oracle VM Server 3.

Dear all,

JBoss® Operations Network (JBoss ON) a key component of Red Hat’s JBoss Managed offerings, provides built-in management and monitoring capabilities to effectively administer all of your JBoss application environments, helping you improve operational efficiency, reduce costs, and ensure a positive experience for your end-users.

If your company or you are based in BeNeLux and are interested by this presentation, just let me know and I will try to arrange a Meeting for you.

Ref : http://www.jboss.com/products/jbosson

BR

Frederic

Despite what some stats may say, my biggest contribution to GNOME is not in bugs or code but in the organization of beer related events!

So I'm pleased to announce that, like each year, we'll have a GNOME Beer party on the Saturday night of FOSDEM (4th Feb). People seemed happy of the location of last year, so we decided to stay at "La Bécasse" in the city center. Feel free to add yourself to the wiki if you are planning to attend.

See you at FOSDEM!

Here is the second batch of interviews with our main track speakers:

• Michael Meeks (LibreOffice)
• Carsten Munk (mobile Linux)
• Kir Kolyshkin (OpenVZ)
• Robert Dewar (commercial success)
• Carsten Haitzler (EFL)

read more

 The number of (Ubuntu) servers I have is growing steadily, and with it approaching 10 (including VM's), I'm starting to look for a solution to manage security upgrades beyond the apt-get update && apt-get upgrade mantra.

What are other people using for this?  I know RH Satellite/Spacewalk is nice but it's only for RHEL-derivates.

Already looked at unattended upgrades, but I'd like something with a little more options and centrally managed (I have redundant web frontends so they should be upgraded sequentially).

Are there any standard solutions out there (not landscape)?  Or should I script it from scratch?

It seems that Ubuntu 11.10 server defaults to NFSv4, which probably makes sense.

But I'm having problems with ownership on the clients.  Both my server and clients run 11.10 and authentication is done with OpenLDAP (so UID/GID's are consistent).

On the server, I do see the correct persmissions on my home directory: UID/GID 2000.

But on the client, this gets mapped to some none-existing uid:

drwxr-xr-x 13 4294967294 4294967294 4096 2012-01-15 01:18 .

I guess the answer will have something to do with idmapd (which is running on the server), but any pointers are welcome!

Il était une fois, un homme qui n'était ni vraiment bête, ni complètement intelligent. Comme il n'était pas tout à fait bête, il se refusait de croire ce qui n'était pas logique à ses yeux. Et comme il n'était pas très intelligent, il ne croyait pas grand chose.

À cette époque, on disait beaucoup que la terre était ronde. Notre homme n'étant pas complètement bête, il se posa beaucoup de questions, comme: « Si la terre est ronde, ceux qui sont en bas ne devraient-ils pas tomber ? » ou « Au fur et à mesure que je m'éloigne d'ici, ne devrais-je pas me mettre à glisser comme sur une pente de plus en plus forte ? ».

Afin de se convaincre, il mesura son jardin très précisément, à la recherche de la moindre courbure. N'en trouvant point, il déduisit fort logiquement qu'il avait été trompé, que la terre était plate.

Il n'était pas très malin mais comme les gens autour de lui l'étaient encore moins, on tomba d'accord avec lui.

Encouragé, il écrivit un livre qui expliquait Ô combien la terre était plate et que comme elle était plate, Ô combien il était important de bien vivre pour éviter qu'un côté soit plus lourd que l'autre et que le monde se mette à pencher.

Des savants vinrent débattre avec lui et exposèrent, avec beaucoup d'équations et un amphigourique vocabulaire, que la terre était une sphère géodésique. Comme c'était compliqué, on décida qu'ils avaient tort.

Notre homme, élevé au rang de prophète, mourut et on le pleura beaucoup. Puis on s'arrêta de pleurer et on continua à enseigner son Livre aux enfants.

Les enfants, ça pose des questions compliquées. En leur disant que tout ce qui était important à savoir était dans le Livre, on s'assura d'éviter d'embarrassantes situations. Comme le Livre était simple à comprendre, tout le monde était content.

Un jour, un enfant devenu adulte embarqua dans un avion abandonné par les savants. Il monta si haut qu'il dépassa le ciel.

Lorsqu'il revint, il expliqua à la foule rassemblée qu'il avait vu la terre de plus haut que n'importe quel homme et que, contrairement à ce que disait le livre, la terre était ronde ! Il en avait même fait le tour.

La foule commença par le traiter de menteur, l'accusa de ne pas avoir été capable de monter si haut et d'affabuler pour se justifier. Voulait-il que les gens se mettent à mal vivre ? On le frappa, le déchira, le lyncha, le démantibula et on accrocha ses membres à un poteau pour montrer à quel point on n'aimait pas les menteurs. Car le Livre l'affirmait: mentir pouvait déséquilibrer la Terre !

Un second enfant, également devenu adulte mais un peu plus tard, fut lui aussi curieux de faire fonctionner cette étrange machine abandonnée. Il monta encore plus haut que le premier enfant et fit deux tours de la planète.

En redescendant, la mésaventure de son prédécesseur lui revint à l'esprit. Devant la foule réunie, il déclara en balbutiant qu'il était monté très haut et que, en effet, la Terre était plate, vraiment très très plate, et qu'il fallait la maintenir en équilibre.

C'était la preuve que les gens attendaient. De foi, la croyance devenait un fait scientifique. On le porta en triomphe, lui offrit du vin et des mets délicats tandis qu'on glorifiait le génie visionnaire de l'auteur du Livre qui, avec les moyens du bord dans son petit jardin, avait réussi à percevoir la Vérité.

A month ago I added BrowserMob to my toolbox. I’m sure I’m the last web-guy in the world to discover BrowserMob (or “Neustar Web Performance”, as of yesterday), but just in case you don’t know them either, it is an online service that provides availability- and performance-monitoring for websites and -applications.

Great stuff, really; create a simple script by providing a URL, choose what datacenters you want the test to run from, set the interval and there you go. After a couple of minutes you can start gazing at charts & reports or check your mailbox for alerts. You can create more complex tests using a JavaScript-based syntax or you can import Selenium-scripts (hello Selenium IDE for FireFox). The free account I started out with offers a substantial amount of pageviews/ month (40.000) that tests can generate.

Possibly related twitterless twaddle:

• Google Webmaster Tools Irony
• Who’s re-baking my cookies?
• Fun with RFP’s: organizing a RAD-race

This year, we will provide two meeting rooms (aka "BoF rooms") in the AW building. The idea here is that they are provided for unplanned or, rather, spontaneously planned and brief meetups.

read more

Ik heb de film 2012 nog altijd niet gezien. Spesjaal om alle Maya-bijgelovigen te jennen, ga ik wachten tot 21 december 2012 om die film te bekijken. Har har har!

Ik ben van plan om die vrijdagavond ineens een filmmarathon te doen, met alleen maar rampenfilms. Een paar vrienden uitnodigen, pizza, popcorn, ambiance en tussendoor een beetje tweeten over de ramp die zich op dat moment afspeelt op het scherm. 't Zou wel leuk zijn als er die avond nog mensen zo'n rampenfilmmarathon zouden doen bij hen thuis en dat we allemaal samen de 2012-believers een beetje pesten.

2012 staat in ieder geval op het programma. Welke rampenfilm mag er zeker niet ontbreken? De ramp moet liefst zo groot mogelijk zijn, dus geen klein brandje, vulkaanuitbarsting of bootongeval. Als het even kan ook geen meteoriet die op het laatste nippertje opgeblazen wordt. Dus liefst films waarin minstens 90% van de wereldbevolking het niet overleeft (een zombie leeft niet dus dat mag ook) en een happy end is niet noodzakelijk.

Ik heb al een lijst met apocalyptische films gevonden maar suggesties zijn zeker welkom!

Dear *,

Here is the definitive JBoss.org schedule for FOSDEM’12 :

Saturday 2012-02-04

Ref : http://www.fosdem.org/2012/schedule/track/jbossorg_devroom

BR

Frederic

The idea of this article came from a colleague of mine. He wrote a first version of the script described below. I found it very useful and asked his permission to re-use it and to write this blog article. Thanks to him! In the mean time, during my researches, I also found that a friend, Didier Stevens, published on his blog the same kind of script but for an AirCap adapter. Mine uses any adapter capable to be switched to “monitor” mode.

All devices have Wi-Fi interfaces (laptops, tablets, mobile phones, consoles, etc) and their operating systems have features to easily manage the wireless networks you connect them to. When you connect for a first time to a new network, most users save the informations for later use (or the system stores it for you without notification). This small database will be used later by the operating system to discover which known network(s) is(are) available and automatically connect to them.

This database may contains a lot of interesting data. Some may reveal private information like your employer, your ISP, where you go to party, to eat, where you go on holidays or which security conference you attended. Why? Simply because networks are often configured with explicit names. Have a look at the screenshots below taken from a laptop running Ubuntu:

(Click to enlarge)

(Click to enlarge)

By default, when a new wireless network is configured, the flag “auto-connect” is enabled. This is the case on Ubuntu, MacOS and Windows 7. What does this mean? Each time you boot your computer or you reconfigure your Wireless card, the device will sent “Probe Request” management frame over the air. This can be compared to a message like “Hey! Network xxx are you there?“. Even if your network uses encryption, all those probes are sent in clear! In Wi-Fi technologies, they are several methods available to detect the available networks or SSIDs:

• Beacon,
• Probe Requests,
• Probe Responses,
• Association Requests,
• Reassociation Requests

“Probe Requests” are very interesting to be captured to detect the SSID’s already configured and used by people. To achieve this, we just need a BackTrack 5, a Wi-Fi network card that supports monitoring mode and some tools. To collect “Probe Requests“, just use the following commands:

  # iwconfig wlan0 mode monitor
  # iwconfig wlan0 channel <i>
  # tshark -i wlan0 subtype probereq

It’s easy but not very convenient! If you keep tshark running a few hours, you could miss data. The purpose of the script is to automate this process and keep some statistics about the detected probe requests (clients MAC addresses and SSID’s). It’s also important to scan all the available channels (1-14) to grab as much SSID’s as possible. This is called “channel hopping” and to achieve this, the script starts a child process which changes the Wi-Fi channel every 5 seconds within an infinite loop. The script syntax is the following:

  Usage: ./hoover.pl --interface=wlan0 [--help] [--verbose] [--iwconfig-path=/sbin/iwconfig]
                    [--ipconfig-path=/sbin/ifconfig]
                    [--dumpfile=result.txt]
  Where:
  --interface     : Specify the wireless interface to use
  --help          : This help
  --verbose       : Verbose output to STDOUT
  --ifconfig-path : Path to your ifconfig binary
  --iwconfig-path : Path to your iwconfig binary
  --tshark-path   : Path to your tshark binary
  --dumpfile      : Save found SSID's/MAC addresses in a flat file (SIGUSR1)

It will dump all detected SSID’s to the console in a completely passive way. No packets are sent over the air from the scanning host! When you kill the script or wake it up via a SIGUSR1 signal, it will dump all detected SSID’s, MAC addresses, packets count and the last time if was seen. The example below shows the result of one day of scan in my neighborhood. 40 SSID’s detected in my area is not bad (I’m leaving in the countryside).

 !! Dumping detected networks:
 !! MAC Address          SSID                           Count      Last Seen
 !! -------------------- ------------------------------ ---------- -------------------
 !! 7E-62-89-9E-C4-E4    Billi-Wifi                             43 2012/01/10 22:15:36
 !! 07-46-6E-4F-61-4E    Réseau de ******                     2732 2012/01/11 16:28:09
 !! 6F-B6-11-2E-AF-74    LA HAGOULLE                             1 2012/01/11 16:17:08
 !! 8F-9F-B1-5B-73-C8    Go-Away-Lamerz                         85 2012/01/11 16:28:09
 !! 00-ED-E1-3A-A9-1C    wifi94                                  6 2012/01/10 18:25:27
 !! E1-28-7F-6A-C6-44    3cles                                   1 2012/01/11 16:17:08
 !! 4E-CD-8A-BD-1C-EB    NOW-X-54                               10 2012/01/10 20:08:02
 !! 0B-8C-A1-1C-BB-51    CRAPS                                5598 2012/01/11 16:28:09
 !! 91-4A-F0-42-A6-63    bbox2-****                              1 2012/01/11 10:48:49
 !! 0B-A7-51-ED-E1-FA    SpeedTouchD4288C                        2 2012/01/11 16:17:08
 !! C09-C2-23-89-2D-E9   ISFS                                    4 2012/01/10 18:12:25
 !! CE-7C-B6-58-39-D3    HAYEZ                                   1 2012/01/11 10:48:49
 !! 44-45-60-E6-61-1B    Guest                                   1 2012/01/11 16:17:08
 !! 0B-A7-51-ED-E1-FA    bbox2-****                              8 2012/01/11 16:15:11
 !! 09-C2-23-89-2D-E9    biblio                                  1 2012/01/11 10:48:49
 !! CE-7C-B6-58-39-D3    free-hotspot.com                        2 2012/01/11 16:17:08
 !! 37-F3-65-28-35-0C    123EURO                                 1 2012/01/11 16:17:08
 !! E4-8F-02-9B-E8-3C    FREE_DELIRIUM                           1 2012/01/11 10:48:49
 !! 6E-2C-81-CE-13-E3    bbox2-****                              4 2012/01/10 18:25:27
 !! E9-4A-D6-4F-72-0C    chateau_magique                         1 2012/01/11 16:19:07
 !! A4-B4-B3-FC-B0-75    WiFi_FD                                 1 2012/01/11 16:17:08
 !! E3-9E-A3-9F-A1-F7    TP-LINK_******                        519 2012/01/11 16:10:51
 !! DA-6C-E2-D8-D8-A7    bbox2-****                              6 2012/01/10 18:25:27
 !! 03-94-41-21-6C-C2    bbox2-****                              3 2012/01/10 18:25:27
 !! 27-E3-1F-61-5A-69    linksys-n                               1 2012/01/11 10:48:49
 !! 81-8A-48-1B-DF-20    Philips WiFi                            1 2012/01/11 10:48:49
 !! 55-C3-BE-F9-63-60    SpeedTouch******                        1 2012/01/11 16:17:08
 !! F0-3D-CC-D3-16-A4    blanmont                               27 2012/01/11 16:28:09
 !! 7A-19-39-BC-3B-A6    chouchou                                1 2012/01/11 10:48:49
 !! 7E-62-89-9E-C4-E4    belgacom                                1 2012/01/11 10:48:49
 !! 07-46-6E-4F-61-4E    Réseau UAH                              4 2012/01/10 18:25:27
 !! 6F-B6-11-2E-AF-74    dlink                                   5 2012/01/11 10:48:49
 !! 8F-9F-B1-5B-73-C8    sagem-****                              1 2012/01/11 16:17:08
 !! 00-ED-E1-3A-A9-1C    bbox2-****                              1 2012/01/11 10:48:49
 !! E1-28-7F-6A-C6-44    bbox2-****                              2 2012/01/11 10:48:49
 !! 4E-CD-8A-BD-1C-EB    QuickWiFi                               1 2012/01/11 16:17:08
 !! 91-4A-F0-42-A6-63    bbox2-****                              1 2012/01/11 16:17:08
 !! 81-8A-48-1B-DF-20    linksys                                14 2012/01/11 16:19:07
 !! 27-E3-1F-61-5A-69    WiFi_6E                                 1 2012/01/11 16:17:08
 !! 82-94-05-84-30-ED    Sitecom                                 1 2012/01/11 16:17:08
 !! Total unique SSID: 40

Note: the MAC addresses have been randomized using the MAC Address Generator.

That’s all for the technical part. Now that you have a list of MAC addresses and SSID’s, what can you do with them? How can this script be useful from an attacker perspective?

First, use this as a “presence detection” mechanism. You can track the presence of people in a specific area. Being at home, I could detect when my neighbor is back at home and uses his laptop. Same for companies. Behind outside, you could detect the presence of employees in the office. More your antenna is powerful more you will be able to detect activity from a long way. Then, the detected SSID’s could help you to learn a lot about your potential victim. The goal is to “put a face” on the MAC address. You can learn the type of device/ISP they use. You can learn about the habits (and later to perform social engineering). hotel SSID’s, restaurant SSID’s etc.Some people defines SSID’s with personal data: pet names, street addresses, nick names. Always interesting stuff… If you know that your victim booked an room in a specific hotel, it’s a step forward to asking him to click on a rogue document coming from this hotel. But that’s another story!

The script is available here.

After a successful first and second edition of LOAD the crew decided to organize a third edition.

You are invited to submit a proposal to participate in the 2012 Linux Open Administration Days in Wilrijk, Belgium on March 31st - April 1st.

The Linux Open Administration Days is a conference focusing on Linux and Open Administration, we are trying to fill a gap for System Engineers and Administrators using Open Source technologies.

read more

Dear all,

Again an interesting talk. This one will be about DeltaCloud API.

Michal Fojtik will be presenting in Virtualization and Cloud Devroom on  Saturday 2012-02-04

15:00-15:25 Open Clouds with Deltacloud API   (room : Chavane)

Here is the abstract :

Start an instance on an internal cloud, then with the same code start another on EC2 or Rackspace. Deltacloud protects your apps from cloud API changes and incompatibilities, so you can concentrate on managing cloud instances the way you want. Deltacloud API prosper from natural open-source evolution and use community driven API design instead of vendor enforced one.

Ref : http://deltacloud.apache.org/

Ref : http://www.fosdem.org

BR

Frederic

FOSDEM 2012 is almost upon us, and we're looking for motivated people to help us make it a success again. If you've visited FOSDEM in the past, you've probably seen our enthusiastic army of volunteers that helped us make FOSDEM a pleasant experience for all our visitors. If you want to be a part of this great team, here's your chance to sign up!

read more

This NetworkManager thing in Ubuntu is horrible. Well; its not too bad actually but the commandline documentation sucks very very hard!
A quick brain dump after my questing for anyone dealing with this.. excuse my messy text.

A Google search for “Ubuntu NetworkManager Commandline” offers help disabling the NetworkManager; installing it (help.ubuntu) and refers you to the nmcli command. Which is just great help. No info to configure it without using the sodden graphical interface to do whatever.
The info about installing was particularly useless to me since it was already installed though the info about where the config files are and reference to nmcli was moderately interesting. Pity there is no info about those config files; anywhere. So .. I happily dicked around with generating configs on my laptop and using those on the headless xbmc box.
Right; so far my frustration; next up: trying to provide some meaningful info..

You probably won’t have to fuss too much with this as “Network Manager auto creates connections on a best effort base” though sometimes, and certainly on headless machines, you just want a fixed IP..

According to the help.ubuntu info, the configurations are in gconf or /etc. I did not find any of that data in gconf; but did find it in /etc

/etc/NetworkManager/system-connections/

the config files go here.

The directory contains all your network configs and NM promises to try to choose the best possible network connection.
Configuration files are owned by root:root and have 600 rights. They are formatted as follows; you will need to edit the UUID in the config files, check the nmcli part below.

root@Benedict:/etc/NetworkManager/system-connections# cat Wired\ 2.44

[802-3-ethernet]
duplex=full

[connection]
id=Wired 2.44
uuid=6a6e191a-4a8b-47ea-bc38-ef8b98748281
type=802-3-ethernet
timestamp=1318578920

[ipv6]
method=ignore

[ipv4]
method=manual
dns=192.168.2.1;
addresses1=192.168.2.44;16;192.168.2.1;

The “addresses1=192.168.2.44;16;192.168.2.1;” is formatted as IP;Netmask;Gateway

or

root@Benedict:/etc/NetworkManager/system-connections# cat Auto\ C

[connection]
id=Auto C
uuid=b6006760-005b-4fc7-b29a-f3565b6fdd8e
type=802-11-wireless
permissions=user:gert:;
timestamp=1320427786

[802-11-wireless]
ssid=C
mode=infrastructure
seen-bssids=00:18:aa:aa:aa:aa;
security=802-11-wireless-security

[802-11-wireless-security]
key-mgmt=wpa-psk
wep-key-flags=1
psk-flags=1
leap-password-flags=1

[ipv4]
method=auto

[ipv6]
method=ignore

For more info about getting your wireless network up; do a google search; the info is out there!

Next up: nmcli

At any rate; nmcli (command-line tool for controlling NetworkManager) wont be much help beyond listing data as “It is not meant as a replacement of nm-applet or other similar clients. Rather it’s a complementary utility to these programs.” You do need it to at least find out more about the connections, UUIDs and what NM is doing..

“nmcli con” lists the available connections

# nmcli con
NAME UUID TYPE TIMESTAMP-REAL
Wired 472a4a85-b432-446c-a704-c7df7b7f5e3e 802-3-ethernet Wed 11 Jan 2012 12:18:48 AM CET
Wired connection 1 472a4a85-b432-446c-a704-c7df7b7f5e3e 802-3-ethernet Wed 11 Jan 2012 12:18:48 AM CET
C e99af4da-5c7a-495e-b1ec-45c81519ad32 802-11-wireless Wed 11 Jan 2012 12:18:48 AM CET

The wired connection is my fresh; hand made connection; C wireless network was configured using the gnome interface; the “Wired connection 1″ was created automatically by NM. Your new connection wont show up however without the right UUID, You need to copy the UUID for the connection you want to use to the config file.

Restarting the networking will choose the configuration file instead of the best effort config.

# /etc/init.d/networking restart

After restarting the connection, the best effort “Wired connection 1″ vanished. I havent found anything about how to influence what connection is used when the best effort choice isnt the right one, but only need the one so I didnt really look either

I hope this helps

Dear all,

Well, it seems for me this year is a FOSDEM  millesime.

Apparently, there is an interesting talk about Aeolus.

Francesco Vollero will be presenting in Virtualization and Cloud Devroom on  Saturday 2012-02-04

14:00-14:25 the Aeolus Project (room : Chavane)

Here is the abstract :

Dear all,

BoxGrindier is going to be present @ FOSDEM’12.

Marek Goldmann will be presenting in the JBoss.org dev room on Saturday:

11:00 – 11:55 BoxGrinder : Grind your appliances easily

Here is the abstract :

BoxGrinder is a set of tools that help you grind out appliances: preconfigured disk images with the operating system and requisite software ready to run on a selected virtualization platform. With a simple text definition file and a single command BoxGrinder will build a lean appliance from scratch, convert it to a target format, and deliver it to your chosen infrastructure.

Virtualization has become almost ubiquitous in modern scalable infrastructures, with traditional dedicated hardware setups being replaced by multi-tenanted virtual environments. This change facilitates many of the beneficial properties of Cloud Computing, key amongst which is the ability to create small, function-specific appliances that enables system components to scale independently.

BoxGrinder addresses the key problem of specifying and building such appliances. It manages complexities such as software installation, dependency resolution and remote service interactions on your behalf. Existing methods can an arduously convoluted, with poor repeatability and performance characteristics; wasting time and resulting in slow and bloated appliances.  Such factors negate many of the motivating factors for embracing Cloud computing, and this is an obstacle that BoxGrinder is specifically designed to overcome. Coherent and simple to specify and run; yet fast to build and easily customized to any desired level of complexity.

In this session we will cover the basic concepts of BoxGrinder, with discussion of techniques and use-cases that illustrate how best to utilise BoxGrinder’s powerful feature-set. Finally, we will work through a live example, from text definition to a running virtual appliance.

Ref : http://boxgrinder.org/

BR

Frederic

(Source: rylanclayne.com)

Today almost all organizations outsource some of their IT projects to third party partners. Due to the ever changing landscape in information technology, it is virtually impossible for an organization to have internal knowledge in all domains of technology. The web presence is maybe one of the most domain where projects are outsourced to “web agencies”. Today, organizations must have an on-line presence and look attractive to customers, investors and more. You know all what we put under the term “Web 2.0″. Hélas, still today lot of web agencies don’t have a clue about security or do not give the right priority to security.

The recent attack  against the Arcelor Mittal website is a very good example! They were compromised by Anonymous Belgium and some data were posted on pastebin.com. My goal is not to debate about the Anonymous Belgium’s actions (which remains illegal in Belgium as in most countries). Hacktivism has pro and cons.  But when data are posted, it’s always interesting to have a look at them to learn more about the attack. In this case, the website Achille’s hell was a Perl script:

http://www.arcelormittal.com/distributionsolutions/prg/selfware.pl?id_sitemap=1

http://www.arcelormittal.com/fce/prg/selfware.pl?id_sitemap=1

http://www.arcelormittal.com/automotive/prg/selfware.pl?id_siremap=1

http://www.arcelormittal.com/distributionsolutions/prg/selfware.pl?id_sitemap=1

First, is the script common or is it part of a well-known CMS? Google gave me the answer. The query “inurl:selfware.pl” returned only 2960 hits! Most of them referring to Arcelor Mittal websites but also other domains of activity:

• arcelormittal.com
• arcelormittalgent.com
• constructalia.com
• prepaintedmetal.;eu
• prepaintedmetalacademy.eu
• prelaque.com
• ziekenhuiswaregem.be
• seniordepartment.be
• prelague.com
• aep-group.eu

Of course, the same script makes other websites vulnerable to the same SQL injection. No need to fire your sqlmap, just by adding a single-quote (‘) to parameter, you get this error:

  Software error:

  SELECT id_sitemap, s_type FROM sitemap WHERE base = 14' AND active = 'Y' ORDER BY s_order LIMIT 0,1
  You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near '' AND active = 'Y' ORDER BY s_order LIMIT 0,1' at line 1 at libs/selfware.pm line 591.

By checking the primary name servers and whois registration data, it’s easy to discover that all the sites mentioned above were developed by a unique web agency located in Belgium. I won’t give the name here (such companies are sometimes more reactive on the legal aspects instead of fixing their crappy code). And my goal is not destroy their image, they are big enough to do it by themselves!

Let’s put the technical stuffs aside now. What are the conclusions of this story? If you outsource some (web) development tasks to an external partner, don’t forget that YOUR name will in front of the stage! The data breach had a big impact for Arcelor Mittal. Their name was present in all media (social and classic). It’s up to you to take the appropriate measures to avoid this situation. Everything can be outsourced but not your responsibility. The written code is used by your customers or team-members and process your data! How to address this issue?

Scenario 1: You delegate the full development life-cycle to your partner. In this case, you must implement controls to verify the compliance with the original requirements during the complete development cycle.

Scenario 2: You delegate the development part to your partner but you perform the compliance controls (code review, penetration testing). This can be done internally or by a third-party partner.

In both cases, a close relation must be established with the partners. Finally don’t put all the eggs in the same bag: Some people are good developers, others are skilled system administrators. Is it a good idea to host your websites on an external server maintained by your web agency? Do they apply patches? Do they monitor the servers? Do they keep an eye on the logs? Like cloud services, the primary goal is often costs reduction. But it must be properly implemented otherwise, the costs could be… worse! Like a simple SQL injection in crappy code…

LPI, BSDCG and TYPO3 will again provide FOSDEM attendees with the possibility of taking their respective certification exams.
Head over to this page for further details.

While Johnny was explaining to the rest of the world how CentOS 6.1 and 6.2 were released, I received quite some questions about the QA tests and how they were performed. Well, let me explain in some words how it's now organized. Previously, there was only a Tests Matrix that was shared between the QA team members : each member of that group had access to the QA bits, could download/rsync the complete tree (with ISO images too) and do his tests, and then reported the results in one way or the other (irc, mailing-list). Of course it didn't scale out very well. Too much manual intervention, and when someone was busy with personal (or work related) issues, no feedback was coming back to the CentOS devteam.

So during Fosdem 2011, I had a meeting with Karanbir to see how we could solve that issue and put automation in the QA loop. We dedicated some (old) machines to be used only for QA, and in a separate VLAN. Basically, here are the steps from the built bits to the QA reports.

• The CentOS buildfarm (using the newly build system called 'reimzul' and using beanstalkd as a queuing system) pushes automatically each new tree to the dedicated QA hardware
• There is a rsync post-xfer script that is launched from there that also uses beanstalkd and some workers (so we can scale out easily if we add machines)
• Each built and pushed tree/ISOs set has its own BuildTag (that is used to identify what was tested and when)
• Some tools (hosted in an internal Git repository) are then used to deploy some Virtual Machines (actually a mix of BareMetal and VMs : blade/Virtual Box/Xen/KVM) and send a report if the "deploy VM step" failed (VMs are installed through ISO/pxe boot/virt-install through http/ftp/nfs methods)
• A test suite (that we call the t_functional stack) is then copied from the local git repo to those newly deployed machines and each test is then ran. From that point a report is then automatically sent to the QA mailing-list so that people can see the results, while the full log is available on QA head node.

The fact that we use two separate git repositories (one for the deploy/provisioniong functions and another one for the tests themselves) was really a good thing, as it permitted some people to include their tests in the t_functional stack. For example , Athmane did a great job writing/fixing some tests used for 6.1 and 6.2.

More informations to come later about how you (yes, *you*) can participate and contribute such CentOS QA auto-tests !

Dear all,

I am going to do an introduction about “JBPM 5.2” in Brussels in January 2012.
If your company or you are based in BeNeLux and are interested by this presentation, just let me know and I will try to arrange a Meeting for you.

Ref : http://www.jboss.org/jbpm

BR

Frederic

AddToAny, one of the most popular sharing-widgets around, has had 3rd party tracking by Media6degrees for quite some time already. I wasn’t too happy about that, but it did have the no_3p option to disable this “functionality”. Half a year ago however AddToAny was acquired by Lockerz.com and it now includes tracking by Lockerz.com which cannot be turned off and does not check for navigator.doNotTrack either.

I’ve contacted the developer (Pat’s a swell guy, really) and he answered he would look into honoring the DoNotTrack header, which he wrote he’d love to include in Q1 somewhere. In the mean time, if you have AddToAny on your site, you can already hide the Lockerz “Earn” tab. And if you’re on WordPress, you could install (or upgrade) WP DoNotTrack, which I’ve updated to stop the Lockerz tracking (make sure lockerz.com is your blacklist).

If there’s a Drupalista out there that uses AddToAny and would like to stop Lockerz tracking; I’d be happy to co-author a Drupal DoNotTrack module, do get in touch!

Possibly related twitterless twaddle:

• Why your WordPress blog needs DoNotTrack
• AddToAny: removing the “spy” from the share-ware
• AddToAny removed-from-here