Bernardo Donadio

DevXperience: quando a baleia azul encalha

Tue, 05 Sep 2017 03:00:00 -0300

Fiz uma talk no DevXperience 2017 sobre os eternos bugs do Docker. Dê uma olhada!

Se preferir, pegue os slides da palestra em PDF para acompanhar melhor.

Lembre-se que apesar dessa ser uma talk falando de bugs do Docker, e no mais sendo negativo sobre a ferramenta, eu uso o Docker diariamente e advogo pela sua aplicação onde este já funciona bem. Containerização já revolucionou a computação, mas ainda temos que aparar algumas arestas para que possamos aplicá-la em qualquer lugar.

Feedback é muito bem-vindo!

Writing a CV for an IT position

Sun, 03 Sep 2017 03:00:00 -0300

Here’s a quick list of Dos and Don’ts when writing resumés for an IT position.

I’m an engineer at Stone Payments, a brazilian credit card acquirer, and a big part of our workforce is composed of IT people. Also, I’ve entered a team in the company that was quickly growing, and therefore had to interview a lot of prospect new colleagues. While the People Team (that’s how the call the HR here) definitely does a great job of helping us to decide about the interviewed’s character and attitude, it cannot help us when determining their technical skills. Therefore, we, engineers, also have to do interviewing.

The result is that I noticed that a breathtakingly large sum of programmers/sysadmins don’t know how to write a proper curriculum for their area, relying instead on general tips found on job sites or nothing at all. So I decided to try to help those people on how to express their skills that sometimes passes under our radar, possibly losing valuable candidates in the process.

These tips not only will help you to write a curriculum that is great to read, but will also show you’re capable of prioritizing relevant information and make your CV stand out from the rest. We receive a lot of noise, and is not always easy to scoop up good applicants from the pile of CVs we receive daily. Help your prospect employer to choose you!

So, here’s it:

DO

Organize your jobs in reverse chronology, most recent first. List a few bullet points on each one about what you did. Write when you entered it and when you left it. Don’t write why: we will ask this in the interview.
Tailor your CV to show the most relevant skills to each prospect employer.
Troughly organize your resumé, and it’s probably better not to use Microsoft Word for this task. If you can do it well with Word, great. I personally cannot, and recommend to use a tool that makes a very clear distintiction between content hierarchy and formatting. We like and recommend LaTeX. Is very easy to distinguish a document made with LaTeX and shows you know how to deal with professional documents. It also shows you know how to treat documents as code, which is very important for a devops-cultured company.
Make a list of technologies/languages you’ve already worked with. Even though we don’t hire for knowledge on specific technologies (we prefer to teach), it’s easier to know what your expertise background is. It also helps when scanning trough a pile of CVs.
Put the links to (if you have): your Github, your blog, your Twitter. Github is a must for a programming position, keep a rich, well-organized and interesting Github profile. A blog is a plus, it shows you know how to express yourself and are communicative. If you normally tweet about your job or IT-related hobbies, also put your Twitter account. Be aware, thought, that most IT companies scoop their employees’ social medias for antisocial behaviour, like constant trolling, prejudice and, of course, crime-related talks.
Put the city you live on.
Put hobbies related to IT. We love seeing that the applicant is really passionate about the area they work in.
List your personal projects (IT-related) if you have any. In the interview itself we may ask for non-IT-related projects to get to know you better.
Use the PDF format for the resumé. In IT, people use a pletheora of different operating systems and office suites: the only format that looks uniform and is well parsed in all those is PDF.
Put your name and date in the filename.
Put a cellphone contact. This makes easier if we have to reschedule the interview if Skype crashes, Hangouts don’t open or any kind of comm technology don’t work (which is kind of the norm, unfortunately).
Write your email address. It makes easier to find you if we print the PDF to read with colleagues and then happen to not find your email in our mailbox.
Put your major degree if you have one, but my company normally doesn’t care about it. It looks pretty, though.
Put your technology certifications. We value those much more than a college degree.
Put the talks/workshops/demos/courses you may have ministered. It shows you can pass along your knowledge.
Describe the area you love the most in IT. It may be helpful to place you better on the team.
If your name isn’t generally associated with your gender, or you’re transgender, specify how you would like to be treated to avoid tight spots on the interview. Specially in languages like portuguese, it’s very hard to talk to people in a completely gender-neutral way. Be transparent and tell the employer how you like to be called. At companies that respects diversity, this will be very well received and both parties will have a nicer time when meeting in person.
Specify which idioms you know and how well you speak them (novice, intermediate, fluent, native or something of the sorts).
Use a common sans-serif font for titles, and a common serif font for the body. Never more than two fonts.
Put volunteer work you may have done: it shows you have energy and prone to kindness, exactly the kind of people we want around us.

DON’T

Put a photo: most IT people are ugly (myself included). Nobody likes to see that, haha. Also, it may trigger some racial bias (even though we try to avoid, it’s very hard to escape from it).
Write Curriculum Vitae, Resume or anything of the sort on the resumé. The title is your name.
Put your address: we really don’t care which part of the city you live, as long it is the same city we want you into.
Put your high-school name.
Put any kind of religion/politics views. This also helps the triggering of our unconscious biases, and may even show you don’t know how to separate your personal life from your work life. If your views explicitely forbids you from doing something work-related, like working on saturdays for example, mention it on the interview.
Use a childish font like Comic Sans. It makes you look immature.
Send the resume in a .doc, .docx, .odt or anything like that.
Put the phone contact of your mom, SO, or anything like that.
Use a stupid email name you created when you were a child, or from a prehistoric ISP domain.
Put your Facebook. Facebook is much more intimate than Twitter. We really don’t care about your interactions with your exes.
Put your marital status. We may ask it on the interview, but is just like a trivia to break ice. It’s not relevant to the job.
Put your age. This is really specific to IT. We really don’t care. The only exception is if you’re a minor, so there’s need to be a different kind of contract, so in this case mention it.
Write banal hobbies: like Netflix or sleeping. We normally ask about hobbies and things you do to space out in the interview and it’s completely OK to answer something like this to break the ice, but this doesn’t belong in the document.

I hope these tips might help you on presenting yourself better. I will provide further tips soon on how to behave in the interview itself, another challenge on IT that isn’t quite what other job offers presents.

Stay tuned.

P2K.co: Send Pocket articles to your Kindle

Mon, 28 Aug 2017 03:00:00 -0300

My Kindle was normally underutilized until I found this great freemium service. If you have a Kindle, you must read this!

Maybe Amazon is really bad at promoting books on their online store (I sincerely think they need to design their interface better and rethink their recommendation algorithms), maybe I’m just too picky. Anyway, the fact is that I bought a Kindle Paperwhite and have really not used it so often as I would like because of a simple problem: the things I wanted to read the most in a cool paper-like interface we’re not on Kindle Store, and Amazon doesn’t have a great open ecossystem where anyone can create apps for their devices.

So I went in search for a good service that would somehow link my bookmarks of assorted Medium articles that I would like to read later on but never reminded that I saved them in the first place. I found the excellent p2k.co service by Emir Aydin and discovered that it can do exactly this!

Basically, p2k.co interacts with the Pocket API to retrieve your bookmarks, generate a custom AZW file with the content of those bookmarks plus a great index and Archive/Like buttons, and then send it directly to your Kindle trough your Kindle mail - which gets synced ASAP to your device in a push-fashion.

The great of it is that P2K aggregates all articles you bookmarked in the day and then send it at a specific time, and I found it great to do so just before the time I normally go to sleep, so I can catch on to the subjects I found interesting trough the day but had not the time to read.

It works just great with the Free plan, but I liked the service so much that even though I really don’t need most of the paid features, I subscribed to the USD$5,00 Plantinum plan to support the development.

If you’re anything like me, you’ll love this great service.

Redismoke: a Redis replication smoke test

Tue, 11 Jul 2017 22:00:00 -0300

I’ve written a small utility to smoke test a Redis replication set. It’s a simple Python script that should be run as a daemon to test if writes to a Redis master are being correctly replicated to the slaves.

The necessity of such script arised from the fact that the Redis Sentinel, tool that Redis uses to implement a kind of raft, isn’t very reliable and has the bad habit of stop working without much notice.

This utility is the work of a single afternoon so far, and it’s being currently worked on to easily integrate with a SolarWinds instance, just as it may be extended to work with other monitoring tools.

You can find the code in the Github repository below:

https://github.com/stone-payments/redismoke

Patches are welcome. The code is distributed under the MIT license. The next features that I will implement will be:

Unit testing (sure, test the test all the way down)
SolarWinds format output
Single run
Separate the CLI from the lib code

Maybe I will write someday about the hell that was creating an idempotent Ansible role for Redis…

Authentication in the cloud

Tue, 04 Jul 2017 02:00:00 -0300

Authentication is hard. How to deal with it in a sane way in a cloud environment? There are two approaches to follow, each with its up and downsides, but which one is the best for your environment?

Foreword

If you’ve got more than a handful of systems, there’s the inescapable need to find a way to uniformize the authentication accross those. However, this is not an easy feat if you have the relatively common need of minimizing downtime to the maximum. So, not only you need to keep bad people out, you also need to always allow good people in, consistently. This decision should always be possible of being executed.

Therefore, you have two main approaches to this issue: centralized and distributed decision. This is a logical issue, not technological one, and is the first choice you must make to go down this path. Each comes with its own pros and cons, but there’s definetely one that fits best your environment. There’s, however, no one-size-fits-all. It ultimately depends on your personal and organizational needs.

From now on, I will present a high-quality example from each of those two domains. Those are battle-tested solutions, used by big-companies like Red Hat, Facebook, Netflix and so on. With these initial informations, I hope you will be able to at least have an initial idea of what you want, and will be able to kickstart your own research.

Also, I will be using in this article the term “authentication” to mean both authentication and authorization, as there’s no need in this article to treat differently those two problems.

Centralized authentication

Don’t be fooled by the cloud hype these days that preach absolute decentralization and promises-like behaviour everywhere. If you have legacy systems, or your objective isn’t simply serving the biggest crowd of users you can possibly imagine, distributed authentication is a problem you don’t need to have.

In fact, Microsoft proved this with its Active Directory solution. It works really well, and meet all goals of a mid/small typical organization. Yes, integration with it isn’t the biggest of its strengths, but Red Hat solved this problem in the Linux domain for sure with its own product.

I will now present this lesser known product from Red Hat: the Red Hat Identity Management, backed from the FreeIPA open-source project. It works so well that you may be forgiven if you think that is a single, monolithic solution like the Microsoft AD. It is, on the contrary, composed of smaller open-source projects and standardized protocols that are already part of every POSIX system. Hence its easiness of integration: there are already a lot of libraries and tooling ready for it, just as your legacy system can be easily instructed to use it.

Also, unlike those previous tools like a hand-maintained OpenLDAP server or a Kerberos KDC, it is freaking easy to deploy. There’s absolutely no need for, ugh, write LDIF files by hand or anything. It’s just as easy as setting up an AD domain-server, and even easier to connect clients to it.

The FreeIPA stack includes the following softwares:

389 LDAP server
MIT KDC server
ISC BIND DNS server
NTF NTPd server
Dogtag CA & RA server
Apache httpd server

And all of it is centrally managed by the FreeIPA web interface, the FreeIPA CLI or the FreeIPA webservice API. There’s absolutely no need to touch directly any of those server’s config files.

Server-side

Provisioning your first FreeIPA server is a little bit different from most POSIX tools you already used, as you don’t really need to touch any config file. Basically you install the ipa-server package with your package manager and then simply call the ipa-server-install script, passing as arguments all infrastructure details you might find relevant at install phase. Aside from the main domain, you can easily change all those later with the server already up and running.

You can find a quickstart guide in the FreeIPA documentation.

From now on, almost all aspects of the FreeIPA deployment and identity content can be administered trough its web interface. Even a junior sysadmin that never touched FreeIPA before can do it. Very few advanced features may be managed trough the FreeIPA CLI, but even those are meant to be implemented in the web interface as soon as there’s developer power to do it.

Even though the FreeIPA server is centralized by nature, it can be distributed against multiple hot-hot FreeIPA servers, providing you with failure-tolerance in a intrisically centralized infrastrucure. The FreeIPA documentation about deployment recomendations show a nice graphic visualization of the supported deployment architecture intra and inter-datacenter.

Client-side

Just as the server side, you don’t need to configure manually any of the subsystems that comprise a full-featured FreeIPA client. All you have to do is install the ipa-client package and run the ipa-client-install script with minimal arguments, since most of the configurations the scripts already discovers automatically trough the DNS SRV entries added to your DNS systems by the FreeIPA server. Then, you just reboot. Really.

You have now added to the host entry in the FreeIPA server the machine’s SSH server key, just as the FreeIPA client shim was installed in both your SSHd and SSH client on this system, enabling you to trust automatically all other hosts’s key in your infrastructure, since FreeIPA keeps a record of it. Also, all users - as long as they’re authorized to do it - can already log in this system trough SSH with their SSH keys associated to their user entry, just as they can either use a password or even a combination of those with two factor authentication. Everything out-of-the-box.

The system DNS A and PTR entries were already added to the FreeIPA DNS server, so you can reach it just by knowing it’s name. The system DNS entries are automatically updated when it boots up, so you can also use dynamic DNS leases.

If your system provides a service that uses a SSL key, you can register in FreeIPA a SSL key to be automatically provided and kept updated trough its automatic renovations, enabling you to use very short-lived keys to enhance security. Provided that the Dogtag server used by FreeIPA is also used by many comercial Certificate Authorities (CAs) around the world, you can rest assured that the implementation of this x509 scheme is sane and trusted.

Finally, you can provide Single Sign-On for both users and servers with the Kerberos service provided by FreeIPA. Once you’ve called kinit and typed your password (which may be combined with a 2FA token), then you can seamlessly log into webservices, SSH servers and every type of service that already supports Kerberos. This includes Windows. Oh yes, FreeIPA integrates with Windows clients and even can be a part of a Microsoft Active Directory domain.

This is all nicely tied up in the provisioning phase by the ipa-client-install script, and in the runtime phase by the SSSD client, so you don’t need to configure any of those manually.

Disaster situation

The big question in authentication for sysadmins, however, isn’t about all the features that a given solution brings to the table, but what happens when shit hits the fan, everything stops working and you need to act fast. SSSD handles that for you, somewhat.

SSSD, just like Windows, does credentials caching in order to authorize access to the system even if it is unable to contact the server to do it online. Also, the credential cache system acts in order the speed things up and decrease the load on server. If you already accessed the system sometime ago, chances are you be able to do it again in the event of a disaster (connection down, FreeIPA server crashed or is otherwise unavailable).

The keyword in this sentence is, however, chances. The SSSD client can’t authorize you if you already dropped out of its cache, or have never seen you. You don’t carry any authorization information with you other than your password/certificate. The server always has to reach the FreeIPA instance to ask about what you’re are or not allowed to do.

Also, this cache is wiped out every reboot, so in the event of a datacenter-wide power failure, you won’t be able to log in any of your systems before the FreeIPA server’s are up and the network is properly working.

In most companies, this can be easily accepted in face of such administration easiness. In a company like Facebook or something, this is unthinkable of.

Distributed authentication

Basically, Facebook was brought to the conclusion that using a centralized solution with their global scale, with them reaching almost a million active servers, is completely insane and much downtime-prone. Therefore, investigating quite a bit, it was possible to create a solution that met all their requirements with something that was already available off-the-shelf, but a little bit obscure so far.

I’m talking about the OpenSSH certificate system, that Marlon Dutra so brilliantly explained in the Facebook’s Code blog.

This solution provides tight security controls, high availability and reliability and even high-performance with a simple trick: you don’t need to contact a login server at all. All the information needed to authorized an access is contained in the user certificate already, that the user itself provides when it wants to log into the system, and the system being accessed simply checks that the signature is a match with the CA trusted by the system.

There’s no SPOF here. You may even login in a machine that is completely isolated from the network by any reason, since the CA certificate is contained in its image (or programatically rotated every once in a while).

Maintaining such an infrasctructure, though, is a litte cumbersome: properly organizing and keeping a secure CA infrastructure is tough and costly, with no helping hands that a solution like FreeIPA might bring. But it can be done, provided you’ve reached the point where this becomes beneficial in terms of time and cost.

Server-side

Well, there’s no server side. There’s just a secure machine that you use to generate certificates, and that may very well be offline for enhanced security. You need, however, a simple webserver to provide the Certificate Revocation List (CRL) for the certificates that were compromised or otherwise put out of use.

Even this webserver, though, don’t need to be kept up all the time. As long as a client system have a recent enough version of the list, you are completely ok. Since this list is also signed, you may very well distribute it in various points of the network or even have clients to exchange those in a peer-to-peer fashion.

What you might want is a central way to audit those accesses, like a Graylog or Hive instance, enabling you to search across chronological data and find possible abuses or invasions. OpenSSH conveniently logs all informations about the certificate used to log into the system to the journal, therefore rendering the data extraction quite a bit simple. Even though, a log-system downtime isn’t going to impact your ability to access your system: the log is delivered in a best-effort fashion.

Client-side

In the distributed solution, that where’s the magic resides. By instructing your OpenSSH server to authorize users based on their certificate’s information after checking their signature, you can carry with the user all the info you might want.

You will need to configure the client to pipe all of its logs to your logging solution, but this problem was long solved with rsyslog or more modern tools like journalbeat.

The downside is, however, that the authorization logic is partly kept in the client itself: you need a way to change this information when you find necessary, be it trough a configuration manager like Puppet or Ansible (which you should already be doing), or

probably like Facebook does - by cycling the whole VM image.

Disaster situation

As long as the CA and CRL are kept within their expiration dates, you will be able to log into the system without help from any external service. Even completely offline.

Also, there’s no performance bottleneck even with billions of simultaneous logins, after all you have no central place to log into. If you already used a undersized Microsoft AD deployment in a college campus or big company, you know what I’m talking about.

Revoking a key is simple as adding the key sum to the CRL, sign and publish it. The revocation will be distributed as fast as you configured your clients to fetch it. Although not so instantaneous as the centralized solution, it is very plausible to do so in face of the benefits brought by this choice.

The third-way

Netflix liked a little bit of both, and developed another solution called BLESS, which runs a CA by-demand, signing very short-lived SSH certificates (like 5 minutes or so) based on business rules.

While certainly this does solves the performance issue, certainly can’t be used as the only authentication mechanish in their infrastructure, as the BLESS system itself is run in the AWS Lamba service, and we know it is too complex to be easily relied on - just as the recent outage taught us.

Afterword

If there’s one piece of advice you should take from this article, it is that you should plan well your disaster possibilities. There’s very few worse sensations than knowing you’ve locked yourself out from your systems in the middle of an outage other than maybe like… almost completely loosing all of your customer data.

However, you shouldn’t be paranoid to the point you may want to decentralize everything for the sake of it, and find yourself in a unentanglable mess of distributed logic with bad visibility.

When the blue whale sinks

Sat, 03 Jun 2017 19:40:00 -0300

A bug in the Linux kernel is affecting thousands of people for more than 3 years, and so far there’s no complete fix. Continue reading to know what to expect when encoutering this issue (hopefully not) in production.

Let’s get started with the biggest PITA I’ve ever experienced with a software in production, that after more than 3 years of being reported (at least), is far from being completely fixed. This issue isn’t present in the Docker code itself, but rather in the Linux kernel code instead. It affects not only Docker, but any kind of software that uses the Linux network stack to create devices and namespaces frequently like LXC, OpenStack, Rkt, Proxmox, etc…

The tell-talle of hitting this bug is receiving a similar message of this every 10 seconds on the VT/syslog/journal of your server:

unregister_netdevice: waiting for veth1 to become free. Usage count = 1

Regarding the message variation, the interface may be any one that is currently being manipulated by Docker (the most frequent case is the lo interface of the container), and the usage count may be larger than one.

If you’ve already had some experience with multi-thread programming, you can instantly diagnose this as a race condition, and even cringe on the memory of debugging those kind of issues. However, this issue affects so many people for so much time (more than 3 years) that you may think that it already got fixed, right? Nope.

Once the bug is triggered, the situation now is the following: you’re unable to create, delete or change any network device in the whole system, rendering Docker basically useless - as you need to do exactly this to create and delete containers. There’s absolutely no fix to the bug so far, except a few mitigations and the Windows-style workaround: reboot your computer. Seriously.

Worse even is the fact that reproducing this issue is far from straightforward. You hit this bug basically by creating and deleting containers frequently, but the frequency needed to hit the issue varies wildy. Some people simply don’t ever hit the bug, and others get systems frozen by this multiple times a day. The Red Hat kernel (including CentOS) seems to be specially more prone to suffer from this problem, but all other distributions have reports of also being affected. Ironically, the sosreport tool used by Red Hat to collect information and logs on the system in order to diagnose the situation simply doesn’t works after the issue is triggered.

There are, however, partial mitigations to address the issue: you can put your virtual bridge docker0 in promiscuous mode to delay the interface teardown just a little as to not hit the race condition (the promiscuous mode has no other relation to the problem than this), or disable the IPv6 support in the kernel, hence it is more prone to hit the bug than the IPv4 stack.

As of the time of this publication, a fix has already been released to the issue being encountered by hitting the bug in the IPv6 stack, but people are still get the annoying message and frozen behaviour, suggesting that the bug has multiple causes, spread all over the network subsystem of the kernel.

This particular fix was released in this linux 4.8 commit and backported for RHEL/CentOS on the kernel-3.10.0-514.21.1.el7 package on RHEL/CentOS, as you can follow in the RHSA#3034221 (RHN access needed).

A very fearsome, but also very possible, scenario is that if you have a PaaS system that auto-heals (like OpenShift or tsuru), you can unleash a chain reaction by triggering the bug on one system. When the auto-heal function takes care of starting the container on other machines, you can now trigger the bug again on those system. Then the thing grows. Exponentially.

Until this issue is completely fixed, I’m very cautious with using Docker for anything other than stateless applications with at least double redundancy.

Yum repository for nginx with ALPN on EL7/6

Sun, 07 Feb 2016 05:18:00 -0200

Chrome needs ALPN to use HTTP/2, ALPN needs OpenSSL 1.0.2, RedHat will only ship OpenSSL 1.0.1 on RHEL 7, and nginx uses the OpenSSL provided by the system. Here’s the solution to this mess!

UPDATE 2: The module ngx_brotli from eustas was added to the build to provide Brotli compression support. It is a fork from the original ngx_brotli from Google, with changes to build with recent nginx versions. Take a look into the Brotli announcement to understand how it may help your pages load faster.

UPDATE: EL7 was updated to OpenSSL 1.0.2 with the release of EL7.4, which provides proper ALPN support, and therefore there’s ALPN support out-of-the-box now on EL7. However, I will continue to be providing packages for EL7 if someone wants to keep using a built-in OpenSSL from the most recent stable branch. The situation with EL6 remains unchanged and it still needs custom packages to provide ALPN support.

TL;DR: Paste this (if you trust me) in your EL7 or EL6 system:

curl -s https://raw.githubusercontent.com/bcdonadio/nginx-alpn/master/repoinst_mainline.sh | sudo bash sudo yum install -y nginx

Current nginx version: 1.15.4
Current OpenSSL version: 1.1.1

Now, let me explain the situation a little bit. Google defined that after Chrome 51, released on May 31st of 2016, HTTP/2 connections would only be negotiated trough Application-Layer Protocol Negotiation (ALPN) instead of the Next Protocol Negotiation (NPN) used so far in TLS-enabled HTTP streams. The new protocol is great because it avoids some additional trips over the wire to negotiate the security parameters, reducing latency, but it needs to be supported on the server side.

Because of the Heartbleed vulnerability, RedHat is no longer doing backports of OpenSSL and other crucial packages, since on the original version of OpenSSL that was shipped with RHEL6 there was no presence of this vulnerability, but it was later introduced because of a stable backport. This creates the whole problem.

RHEL7 shipped with OpenSSL 1.0.1, and RedHat has no plans to update it - as per this RFE ticket - until RHEL8, which is only scheduled to be launched on 2018. That’s two more years without ALPN support.

Therefore, neither Apache httpd nor nginx packages shipped on either the official RedHat repos, the Fedora EL repos nor even the packages available in the Apache and nginx own Yum repos are able to support ALPN, since they all use the OpenSSL dynamic library available on the target system.

Replacing only the OpenSSL package for a more recent version is not an option, since that would introduce an ABI incoherence between the library available on the system and the one that the applications expect. The application also needs to be rebuilt.

The good news is that since we need to rebuild the application anyway, we can easily instruct it to use a different OpenSSL library, but this comes with a whole new bag of problems: you need to keep track of both the OpenSSL and the application (nginx or Apache httpd) security bulletins, besides having to either create a new Yum repository yourself or building the packages in every system you manage.

As such, I decided to build those packages and make it available to the community, and the packagecloud team was happy to host them.

The packages available at packagecloud are distributed either trough direct download, or trough a GPG-signed Yum repository. Also, I always sign the packages themselves with my own GPG key. Therefore, if you trust me, you can be certain that the package is safe.

Basically the script on the top of this post will do the following:

Ensure curl, sed and pygpgme are installed
Create a repository config for this repo on /etc/yum.repos.d
Install the packagecloud and my own GPG keys on the RPM keyring

Note that this nginx package has a built-in OpenSSL library, and will in no way touch your OpenSSL library already present on the system. Also, the script will not install nginx by itself, since nginx is modular, and you may or may not want to also install its modules.

In order to build those packages, I’ve written some Docker recipes for EL7 and EL6, which are fit for either RHEL or CentOS, and those recipes are also open to scrutiny at GitHub.

I’ve also modified the default packagecloud Yum repo install script to also include the installation of my public GPG key, to ensure extra protection against unauthorized modifications on the package.

I vow to keep this repository updated until the EOL of RHEL7 and RHEL6.

If you liked this, drop me a line in the comments and star the repo on Github!

Freeing myself from my ISP with LTE and a Raspberry: Part 1

Sun, 07 Feb 2016 05:18:00 -0200

I really, really hate my ISP.

That said, let me explain what I wanted. Since my connection is very unstable, and still running on VDSL with cooper wires of the PSTN installed 50 years ago, I decided that a second internet connection was needed. Even when fiber is finally deployed here, it’s still a good idea to have a completely separated connection from a completely unrelated provider, since clusterfucks do happen. Often.

So I was left with some interesting choices (but none of those ideal): pay the only other ISP which has cables around here (and uses DOCSIS over coax, which isn’t really better) which has a ridiculously low transfer cap, get a long-distance wifi provider with low speeds and poor receptivity, or go with LTE. By the title, you already know what I choose. Despite also having some very low data caps, it had two main advantages: in case of heavy rain - which frequently disrupts the cabling - it would still survive, and the fact that I was already paying for it (for my smartphone, duh). All I had to do was getting an additional SIM card and an USB modem. And so I did: the choosen one was the Huwaei E3276.

On MercadoLivre.com.br there was an interesting reflector/enclosure from the company Aquario, which gave me a way to fix the modem in the tower and a protection from the weather. Plus, there may be some little advantage of the reflector in the signal quality, but I’m skeptic. Really, it shouldn’t make noticeable difference.

However, there’s an issue: for LTE to work well at full speed, direct view to the ERB is needed for the LTE modem. Lucky me, my father is a ham radio, and - yup - we have a goddam antenna tower above our roof. The tower provides the view to the closest ERB without obstacles in the Fresnel zone. And yes, I calculated it.

The main obstacle is a tree close to my house, just in the straight line connecting my antenna tower and the ERB. Knowing the height of the tree, the height of the ERB, the distance from my tower to the ERB, the distance to the tree meters and the lowest frequency that will be used, all I had to do is solve for x. Google Earth Pro provided easily with the measurements. An engineering background comes in handy those times, huh?

Then I noticed another problem: USB only extends for 5 meters without repeaters, and the 5 volts quickly fall with a lot of cabling, making the connection unstable or even unusable. The USB host had to be close to the modem, probably in the tower itself. But hey, that’s easy. I just got a Raspberry Pi that I had laying around, and bought a waterproof case to put it inside. To power it, a Power over Ethernet splitter and injector was bought. For the Power Supply, I used an old notebook brick which gives 19V and 2A.

Now I had to transform those less-than-19V in which comes from the Ethernet cable (cooper losses, remember?) into steady, noise-free 5V to power the Pi. For this task, I bought an USB vehicle charger, tore apart, and 3D printed a slightly better case with a good barrel connector (instead of the original ligther connector).

Finally, now that I had all this I thought… fuck it. Let’s add an WiFi adapter and a high-gain antenna. This enables me to get connection from nearby open hotspots, which have the wonderful advantage of not having data caps. I chose the TP-Link T2UH USB 802.11b/g/n 2.4/5GHz adapter. The antenna was an omnidirectional one from a local manufacturer, which advertises 12dbi gain - but again, I really doubt it.

In the next post, I will talk on how I put all those things together, and how I installed it in the tower.

Easing the use of Xen's xe with xeh

Wed, 04 Sep 2013 13:51:00 -0300

At Propus, the company I work to, we’ve been using XenServer to virtualize most of our servers. This is great and practical, except for one little detail: the CLI administration tool, called xe, is a huge PITA to be used on a daily-basis. Also, there’s a GUI administration tool, called XenCenter, but only for Windows. Well, there are some GUIs to GNU/Linux, but those are either abandoned or are completely incapable of doing something useful.

So, I decided to write a simple script to ease some of the repetitive, mundane actions of administering Xen. It’s called xeh (as in XE Helper), and it’s publicly available at gitHub under GPLv2.

https://github.com/bcdonadio/xeh

As it was written in BASH in a single day (TODO: learn Python), there’s a lot to improve. I plan to update it as the use in the company grows and new features are requested or bugs are found. Eventually, I might write it again in other (safer, faster, better, real) language, but for now it’s fulfilling my needs.

The main features are the following:

Remote administration
Automatic SSH tunnel and VNC session creation
Simplified object search interface
Ability to use a predefined list of servers and credentials
Good user input validation
Human readable memory value input
Consistent user interface
Just take care with maybe some loose ends!

Getting an IPv6 connection with SiXXs

Tue, 23 Jul 2013 02:14:00 -0300

Actually no major ISP (at least consumer-grade) provides IPv6 traffic nor addressing on theirs networks in the country where I live, but I’m curious, and couldn’t be left out of one of the biggest changes in the Internet since the first browser. So, I came out looking for tunnel options, and found that SiXXs is providing free tunnels to anyone who wants it and have a reasonable motive. Lucky me, plain curiosity is an acceptable one.

After a little bit of research, I found out that there is a PoP in my country, and it does provide a fairly low latency for one of continental dimensions like mine. This provider is CBTC (brudi01) and it has a ping time of ~45ms from Porto Alegre. So, everything was perfect, except for a little detail: how the hell IPv6 works?

The first thing to notice is that IPv6 is completely independent from IPv4, also you can (and need) a complete different set of rules in your firewall to both. So, you thought that keeping a single set of consistent rules was hard, don’t ya? Try to keep two, for every single computer. You should use ip6tables to handle those IPv6 rules, and try very hard to not mix everything up. Also, you can’t really simply copy the rules from one to another, because, as they have - on a quick look - almost the same options, they’re not perfectly compatible. Mainly regarding ICMP packets, where they’re almost completely different. They even baptized the new protocol with a new (but not very creative) name: ICMPv6.

After being accepted as a SiXXs user, I could choose my tunnel type, and the options were the following:

Static: really not an option for me. I don’t have a static IPv4 address, and SiXXs draw ISKs (a system of credits, to punish bad users) from users who do not keep their static connections up 24x7. As I have a dynamic home connection, with a very noisy desktop, this was simply not an option.
Heartbeat: I’ve chosen this one, as I have complete control of the NAT of my network. This modality doesn’t draw credits if a tunnel stays down, and even add credits if you open a connection every day.
AYIYA: the IPv6 MacGyver, is able to trespass almost every kind of NAT, but it poses an overhead both to the client and the server. To my desktop, this overhead would be negligible, and since I can use Heartbeat (I have full control of my NAT), this would be of little use to me.

For some reason, I needed to wait again for approval of my tunnel request. A couple of hours later I was able to set my tunnel. Surprisingly, the AICCU, daemon that handles all those three types of tunnels, installation through apt-get on Ubuntu was so easy that almost took all the fun on setting the tunnel, except for a couple of details. Just type this and follow the wizard:

# apt-get install aiccu

Done with the installation, you need to understand that you received two /64 subnets even without asking for an additional /48 one. The first one is the tunnel endpoint IPs, which ::1 will be the PoP router, and ::2 will be you. Actually, you can use this ::2 to any kind of traffic, because, as everything in IPv6, it’s globally reachable. But you will not be able to use any of the other IPs in this subnet. They’re assigned to your PoP, and will not reach you.

The second subnet is routed directly to you. Any packet traveling with the subnet prefix of yours (disregarding the 64 least significant bits of the address) will reach you. In fact, when wiresharking the connection while pinging any address of the subnet you will see them. Cool, huh? Well, not yet. Those pings will not be answered, because your device is not configured to answer to packets coming from any of those addresses of the second subnet. In order to your device be able to do so, you must add them to the list of IPs that it will answer to, as following:

# ip addr change <ipv6_complete_address>/64 dev sixxs preferred_lft 0

Repeat the process until all addresses have this bit set to zero but the one that you want as default. If you need to list the addresses added and to know the current default one, do as so:

# ip addr

The addresses marked with the deprecated tag will be the ones with the preferred_lft bit set to 0. As more in the top that are the addresses, the later they were added. Remember that the ones with the deprecated tag will continue to be working normally, just won’t be the default ones for new outgoing connections.

Good, but now you have to make those changes permanent, otherwise, whenever the machine is either rebooted or the sixxs interface is brought down and up again, you will lose the addresses configurations made so far. You can do so adding the configurations you made in a new executable BASH script like /usr/local/etc/aiccu-subnets.sh or any other path, like this:

#!/bin/bash ip addr add <your_public_subnet_prefix>::1/64 dev sixxs ip addr change <ipv6_complete_address>/64 dev sixxs preferred_lft 0

Then you must tell AICCU to execute this file whenever the tunnel is established, appending the following line to /etc/aiccu.conf:

setupscript /usr/local/etc/aiccu-subnets.sh

Finally, the glibc default DNS resolver (getaddrinfo(), also called gai) will be preferring to answer with the IPv6 address of a given domain whenever available. If your tunnel does have a fairly high latency, you may not want this. With the following configuration, the tunnel will only be used when the resolver finds out that the given hostname does only have an AAAA entry, as it will always prefer to answer with the A record.

In order to do so, you must add or uncomment the following line in /etc/gai.conf:

precedence ::ffff:0:0/96 100

Now you’re ready to go rock out with your new IPv6 connection. Don’t forget that you must configure a firewall to the IPv6 stack with ip6tables. You’re now globally reachable and there’s no NAT to protect you!

See ya!