rss.png profile for ebal on Stack Exchange, a network of free, community-driven Q&A sites
Working with Yaml and Jinja2 in Python3


YAML is a human friendly data serialization standard, especially for configuration files. Its simple to read and use.

Here is an example:

# A list of tasty fruits
    - Apple
    - Orange
    - Strawberry
    - Mango

btw the latest version of yaml is: v1.2.


Working with yaml files in python is really easy. The python module: PyYAML must be installed in the system.

In an archlinux box, the system-wide installation of this python package, can be done by typing:

$ sudo pacman -S --noconfirm python-yaml

Python3 - Yaml Example

Save the above yaml example to a file, eg. fruits.yml
Open the Python3 Interpreter and write:

$ python3.6
Python 3.6.4 (default, Jan  5 2018, 02:35:40)
[GCC 7.2.1 20171224] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from yaml import load 

>>> print(load(open("fruits.yml")))
{'fruits': ['Apple', 'Orange', 'Strawberry', 'Mango']}

an alternative way is to write the above commands to a python file:

from yaml import load

and run it from the console:

$ python3
{'fruits': ['Apple', 'Orange', 'Strawberry', 'Mango']}

Instead of print we can use yaml dump:


import yaml

'fruits: [Apple, Orange, Strawberry, Mango]n'

The return type of yaml.load is a python dictionary:

<class 'dict'>

Have that in mind.


Jinja2 is a modern and designer-friendly templating language for Python.

As a template engine, we can use jinja2 to build complex markup (or even text) output, really fast and efficient.

Here is an jinja2 template example:

I like these tasty fruits:
* {{ fruit }}

where {{ fruit }} is a variable.
Declaring the fruit variable with some value and the jinja2 template can generate the prefarable output.


In an archlinux box, the system-wide installation of this python package, can be done by typing:

$ sudo pacman -S --noconfirm python-jinja

Python3 - Jinja2 Example

Below is a python3 - jinja2 example:

import jinja2

template = jinja2.Template("""
I like these tasty fruits:
* {{ fruit }}

data = "Apple"

The output of this example is:

I like these tasty fruits:
* Apple

File Template

Reading the jinja2 template from a template file, is a little more complicated than before. Building the jinja2 enviroment is step one:

env = jinja2.Environment(loader=jinja2.FileSystemLoader("./"))

and Jinja2 is ready to read the template file:

template = env.get_template("t.j2")

The template file: t.j2 is a litle diferrent than before:

I like these tasty fruits:
{% for fruit in fruits -%}
* {{ fruit }}
{% endfor %}

Yaml, Jinja2 and Python3

To render the template a dict of global variables must be passed. And parsing the yaml file the yaml.load returns a dictionary! So everything are in place.

Compine everything together:

from yaml import load
from jinja2 import Environment, FileSystemLoader

mydata = (load(open("fruits.yml")))

env = Environment(loader=FileSystemLoader("./"))
template = env.get_template("t.j2")


and the result is:

$ python3

I like these tasty fruits:
* Apple
* Orange
* Strawberry
* Mango
Migrating to PowerDNS

A few years ago, I migrated from ICS Bind Authoritative Server to PowerDNS Authoritative Server.

Here was my configuration file:

# egrep -v '^$|#' /etc/pdns/pdns.conf 




Α quick reminder, a DNS server is running on tcp/udp port53.

I use dnsdist (a highly DNS-, DoS- and abuse-aware loadbalancer) in-front of my pdns-auth, so my configuration file has a small change:


instead of local-address, local-ipv6

You can also use pdns without dnsdist.

My named.conf looks like this:

# cat /etc/pdns/named.conf

zone "" IN {
    type master;
    file "/etc/pdns/var/";

So in just a few minutes of work, bind was no more.
You can read more on the subject here: Migrating to PowerDNS.

Converting from Bind zone files to SQLite3

PowerDNS has many features and many Backends. To use some of these features (like the HTTP API json/rest api for automation, I suggest converting to the sqlite3 backend, especially for personal or SOHO use. The PowerDNS documentation is really simple and straight-forward: SQLite3 backend


Install the generic sqlite3 backend.
On a CentOS machine type:

# yum -y install pdns-backend-sqlite


Create the directory in which we will build and store the sqlite database file:

# mkdir -pv /var/lib/pdns


You can find the initial sqlite3 schema here:


you can also review the sqlite3 database schema from github

If you cant find the schema.sqlite3.sql file, you can always download it from the web:

# curl -L -o /var/lib/pdns/schema.sqlite3.sql  \

Create the database

Time to create the database file:

# cat /usr/share/doc/pdns/schema.sqlite3.sql | sqlite3 /var/lib/pdns/pdns.db

Migrating from files

Now the difficult part:

# zone2sql --named-conf=/etc/pdns/named.conf -gsqlite | sqlite3 /var/lib/pdns/pdns.db

100% done
7 domains were fully parsed, containing 89 records

Migrating from files - an alternative way

If you have already switched to the generic sql backend on your powerdns auth setup, then you can use: pdnsutil load-zone command.

# pdnsutil load-zone /etc/pdns/var/ 

Mar 20 19:35:34 Reading random entropy from '/dev/urandom'
Creating ''


If you dont want to read error messages like the below:

sqlite needs to write extra files when writing to a db file

give your powerdns user permissions on the directory:

# chown -R pdns:pdns /var/lib/pdns


Last thing, make the appropriate changes on the pdns.conf file:

## launch=bind
## bind-config=/etc/pdns/named.conf


Reload Service

Restarting powerdns daemon:

# service pdns restart

Restarting PowerDNS authoritative nameserver: stopping and waiting..done
Starting PowerDNS authoritative nameserver: started


# dig @ -p 5353  -t soa +short 2018020107 14400 7200 1209600 86400


# dig -t soa +short 2018020107 14400 7200 1209600 86400


Using the API

Having a database as pdns backend, means that we can use the PowerDNS API.

Enable the API

In the pdns core configuration file: /etc/pdns/pdns.conf enable the API and dont forget to type a key.


The API key is used for authorization, by sending it through the http headers.

reload the service.

Testing API

Using curl :

# curl -s -H 'X-API-Key: 0123456789ABCDEF'

The output is in json format, so it is prefable to use jq

# curl -s -H 'X-API-Key: 0123456789ABCDEF' | jq .

    "zones_url": "/api/v1/servers/localhost/zones{/zone}",
    "version": "4.1.1",
    "url": "/api/v1/servers/localhost",
    "type": "Server",
    "id": "localhost",
    "daemon_type": "authoritative",
    "config_url": "/api/v1/servers/localhost/config{/config_setting}"

jq can also filter the output:

# curl -s -H 'X-API-Key: 0123456789ABCDEF' | jq .[].version


Getting the entire zone from the database and view all the Resource Records - sets:

# curl -s -H 'X-API-Key: 0123456789ABCDEF'

or just getting the serial:

# curl -s -H 'X-API-Key: 0123456789ABCDEF' | \
  jq .serial


or getting the content of SOA type:

# curl -s -H 'X-API-Key: 0123456789ABCDEF' | \
  jq '.rrsets[] | select( .type | contains("SOA")).records[].content '

" 2018020107 14400 7200 1209600 86400"


Creating or updating records is also trivial.
Create the Resource Record set in json format:

# cat > /tmp/test.text <<EOF
    "rrsets": [
            "name": "",
            "type": "TXT",
            "ttl": 86400,
            "changetype": "REPLACE",
            "records": [
                    "content": ""Test, this is a test ! "",
                    "disabled": false


and use the http Patch method to send it through the API:

# curl -s -X PATCH -H 'X-API-Key: 0123456789ABCDEF' --data @/tmp/test.text \ | jq . 

Verify Record

We can use dig internal:

# dig -t TXT @ -p 5353 +short
"Test, this is a test ! "

querying public dns servers:

$ dig txt +short @
"Test, this is a test ! "

$ dig txt +short @
"Test, this is a test ! "

or via the api:

# curl -s -H 'X-API-Key: 0123456789ABCDEF' | \
   jq '.rrsets[].records[] | select (.content | contains("test")).content'

""Test, this is a test ! ""

That’s it.

Tag(s): powerdns, sqlite, api
Let’s Encrypt Wildcard Certificate

ACME v2 and Wildcard Certificate Support is Live

We have some good news, letsencrypt support wildcard certificates! For more details click here.

The key phrase on the post is this:

Certbot has ACME v2 support since Version 0.22.0.

unfortunately -at this momment- using certbot on a centos6 is not so trivial, so here is an alternative approach using: is a pure Unix shell script implementing ACME client protocol.

# curl -LO
# tar xf 2.7.7.tar.gz
# cd

[]# ./ --version


I have my own Authoritative Na,e Server based on powerdns software.

PowerDNS has an API for direct control, also a built-in web server for statistics.

To enable these features make the appropriate changes to pdns.conf


and restart your pdns service.

To read more about these capabilities, click here: Built-in Webserver and HTTP API

testing the API:

# curl -s -H 'X-API-Key: 0123456789ABCDEF' | jq .

  "zones_url": "/api/v1/servers/localhost/zones{/zone}",
  "version": "4.1.1",
  "url": "/api/v1/servers/localhost",
  "type": "Server",
  "id": "localhost",
  "daemon_type": "authoritative",
  "config_url": "/api/v1/servers/localhost/config{/config_setting}"


export PDNS_Url=""
export PDNS_ServerId="localhost"
export PDNS_Token="0123456789ABCDEF"
export PDNS_Ttl=60

Prepare Destination

I want to save the certificates under /etc/letsencrypt directory.
By default, will save certificate files under /root/ path.

I use selinux and I want to save them under /etc and on similar directory as before, so:

# mkdir -pv /etc/letsencrypt/

Create WildCard Certificate


# ./
  --dns dns_pdns
  --dnssleep 30
  -d *
  --cert-file /etc/letsencrypt/
  --key-file  /etc/letsencrypt/
  --ca-file   /etc/letsencrypt/
  --fullchain-file /etc/letsencrypt/


Using HTTP Strict Transport Security means that the browsers probably already know that you are using a single certificate for your domains. So, you need to add every domain in your wildcard certificate.

Web Server

Change your VirtualHost

from something like this:

SSLCertificateFile /etc/letsencrypt/live/
SSLCertificateKeyFile /etc/letsencrypt/live/
Include /etc/letsencrypt/options-ssl-apache.conf
SSLCertificateChainFile /etc/letsencrypt/live/

to something like this:

SSLCertificateFile    /etc/letsencrypt/
SSLCertificateKeyFile /etc/letsencrypt/
Include /etc/letsencrypt/options-ssl-apache.conf
SSLCertificateChainFile /etc/letsencrypt/

and restart your web server.




Qualys SSL Server Test)



X509v3 Subject Alternative Name

# openssl x509 -text -in /etc/letsencrypt/ | egrep balaskas

GitLab CI/CD for building RPM

Continuous Deployment with GitLab: how to build and deploy a RPM Package with GitLab CI

I would like to automate building custom rpm packages with gitlab using their CI/CD functionality. This article is a documentation of my personal notes on the matter.

[updated: 2018-03-20 gitlab-runner Possible Problems]


You can find notes on how to install gitlab-community-edition here: Installation methods for GitLab. If you are like me, then you dont run a shell script on you machines unless you are absolutely sure what it does. Assuming you read and you are on a CentOS 7 machine, you can follow the notes below and install gitlab-ce manually:

Import gitlab PGP keys

# rpm --import 

# rpm --import

Gitlab repo

# curl -s '' \
  -o /etc/yum.repos.d/gitlab-ce.repo 

Install Gitlab

# yum -y install gitlab-ce

Configuration File

The gitlab core configuration file is /etc/gitlab/gitlab.rb
Remember that every time you make a change, you need to reconfigure gitlab:

# gitlab-ctl reconfigure

My VM’s IP is: Update the external_url to use the same IP or add a new entry on your hosts file (eg. /etc/hosts).

external_url ''

Run: gitlab-ctl reconfigure for updates to take effect.


To access the GitLab dashboard from your lan, you have to configure your firewall appropriately.

You can do this in many ways:

  • Accept everything on your http service
    # firewall-cmd --permanent --add-service=http

  • Accept your lan:
    # firewall-cmd --permanent --add-source=

  • Accept only tcp IPv4 traffic from a specific lan
    # firewall-cmd --permanent --direct --add-rule ipv4 filter INPUT 0 -p tcp -s -j ACCEPT

or you can complete stop firewalld (but not recommended)

  • Stop your firewall
    # systemctl stop firewalld

okay, I think you’ve got the idea.

Reload your firewalld after every change on it’s zones/sources/rules.

# firewall-cmd --reload



Point your browser to your gitlab installation:

this is how it looks the first time:


and your first action is to Create a new password by typing a password and hitting the Change your password button.



First Page


New Project

I want to start this journey with a simple-to-build project, so I will try to build libsodium,
a modern, portable, easy to use crypto library.

New project --> Blank project



I will use this libsodium.spec file as the example for the CI/CD.


The idea is to build out custom rpm package of libsodium for CentOS 6, so we want to use docker containers through the gitlab CI/CD. We want clean & ephemeral images, so we will use containers as the building enviroments for the GitLab CI/CD.

Installing docker is really simple.


# yum -y install docker 

Run Docker

# systemctl restart docker
# systemctl enable  docker

Download image

Download a fresh CentOS v6 image from Docker Hub:

# docker pull centos:6 
Trying to pull repository ...
6: Pulling from
ca9499a209fd: Pull complete
Digest: sha256:551de58ca434f5da1c7fc770c32c6a2897de33eb7fde7508e9149758e07d3fe3

View Docker Images

# docker images
REPOSITORY          TAG                 IMAGE ID            CREATED             SIZE    6                   609c1f9b5406        7 weeks ago         194.5 MB

Gitlab Runner

Now, it is time to install and setup GitLab Runner.

In a nutshell this program, that is written in golang, will listen to every change on our repository and run every job that it can find on our yml file. But lets start with the installation:

# curl -s '' \
  -o /etc/yum.repos.d/gitlab-runner.repo

# yum -y install gitlab-runner

GitLab Runner Settings

We need to connect our project with the gitlab-runner.

 Project --> Settings --> CI/CD

or in our example:

click on the expand button on Runner’s settings and you should see something like this:


Register GitLab Runner

Type into your terminal:

# gitlab-runner register

following the instructions


[root@centos7 ~]# gitlab-runner register
Running in system-mode.                            

Please enter the gitlab-ci coordinator URL (e.g.

Please enter the gitlab-ci token for this runner:

Please enter the gitlab-ci description for this runner:

Please enter the gitlab-ci tags for this runner (comma separated):

Whether to lock the Runner to current project [true/false]:

Registering runner... succeeded                     runner=s6ASqkR8

Please enter the executor: docker, ssh, virtualbox, docker-ssh+machine, kubernetes, docker-ssh, parallels, shell, docker+machine:

Please enter the default Docker image (e.g. ruby:2.1):

Runner registered successfully. Feel free to start it, but if it's running already the config should be automatically reloaded!
[root@centos7 ~]#

by refreshing the previous page we will see a new active runner on our project.


The Docker executor

We are ready to setup our first executor to our project. That means we are ready to run our first CI/CD example!

In gitlab this is super easy, just add a

New file --> Template --> gitlab-ci.yml --> based on bash

Dont forget to change the image from busybox:latest to centos:6


that will start a pipeline


GitLab Continuous Integration

Below is a gitlab ci test file that builds the rpm libsodium :


image: centos:6

  - echo "Get the libsodium version and name from the rpm spec file"
  - export LIBSODIUM_VERS=$(egrep '^Version:' libsodium.spec | awk '{print $NF}')
  - export LIBSODIUM_NAME=$(egrep '^Name:'    libsodium.spec | awk '{print $NF}')

  stage: build
    untracked: true
    - echo "Install rpm-build package"
    - yum -y install rpm-build
    - echo "Install BuildRequires"
    - yum -y install gcc
    - echo "Create rpmbuild directories"
    - mkdir -p rpmbuild/{BUILD,RPMS,SOURCES,SPECS,SRPMS}
    - echo "Download source file from github"
    - rpmbuild -D "_topdir `pwd`/rpmbuild" --clean -ba `pwd`/libsodium.spec

  stage: test
    - echo "Test it, Just test it !"
    - yum -y install rpmbuild/RPMS/x86_64/$LIBSODIUM_NAME-$LIBSODIUM_VERS-*.rpm

  stage: deploy
    - echo "Do your deploy here"


GitLab Artifacts

Before we continue I need to talk about artifacts

Artifacts is a list of files and directories that we produce at stage jobs and are not part of the git repository. We can pass those artifacts between stages, but you have to remember that gitlab can track files that only exist under the git-clone repository and not on the root fs of the docker image.

GitLab Continuous Delivery

We have successfully build an rpm file!! Time to deploy it to another machine. To do that, we need to add the secure shell private key to gitlab secret variables.

Project --> Settings --> CI/CD


stage: deploy

Lets re-write gitlab deployment state:


  stage: deploy
    - echo "Create ssh root directory"
    - mkdir -p ~/.ssh/ && chmod 700 ~/.ssh/

    - echo "Append secret variable to the ssh private key file"
    - echo -e "$SSH_PRIVATE_test_KEY" > ~/.ssh/id_rsa
    - chmod 0600 ~/.ssh/id_rsa

    - echo "Install SSH client"
    - yum -y install openssh-clients

    - echo "Secure Copy the libsodium rpm file to the destination server"
    - scp -o StrictHostKeyChecking=no rpmbuild/RPMS/x86_64/$LIBSODIUM_NAME-$LIBSODIUM_VERS-*.rpm  $DESTINATION_SERVER:/tmp/

    - echo "Install libsodium rpm file to the destination server"
    - ssh -o StrictHostKeyChecking=no $DESTINATION_SERVER yum -y install /tmp/$LIBSODIUM_NAME-$LIBSODIUM_VERS-*.rpm

and we can see that our pipeline has passed!


Possible Problems:

that will probable fail!


because our docker images don’t recognize

Disclaimer: If you are using real fqdn - ip then you will probably not face this problem. I am referring to this issue, only for people who will follow this article step by step.

Easy fix:

# export -p EXTERNAL_URL="" && yum -y reinstall gitlab-ce

GitLab Runner

GitLab Runner is not running !

# gitlab-runner verify
Running in system-mode.                            

Verifying runner... is alive                        runner=e9bbcf90
Verifying runner... is alive                        runner=77701bad

#  gitlab-runner status
gitlab-runner: Service is not running.

# gitlab-runner install  -u gitlab-runner -d /home/gitlab-runner/

# systemctl is-active gitlab-runner

# systemctl enable gitlab-runner
# systemctl start gitlab-runner

# systemctl is-active gitlab-runner

# systemctl | egrep gitlab-runner
  gitlab-runner.service     loaded active running   GitLab Runner

# gitlab-runner status
gitlab-runner: Service is running!

# ps -e fuwww | egrep -i gitlab-[r]unner
root      5116  0.4  0.1  63428 16968 ?        Ssl  07:44   0:00 /usr/bin/gitlab-runner run --working-directory /home/gitlab-runner/ --config /etc/gitlab-runner/config.toml --service gitlab-runner --syslog --user gitlab-runner
Tag(s): gitlab, docker, CI/CD
Encrypted files in Dropbox

Encrypted files in Dropbox

As we live in the age of smartphones and mobility access to the cloud, the more there is the need to access our files from anywhere. We need our files to be available on any computer, ours (private) or others (public). Traveling with your entire tech equipment is not always a good idea and with the era of cloud you dont need to bring everything with you.

There are a lot of cloud hosting files providers out there. On wikipedia there is a good Comparison of file hosting services article you can read.

I’ve started to use Dropbox for that reason. I use dropbox as a public digital bucket, to store and share public files. Every digital asset that is online is somehow public and only when you are using end-to-end encryption then you can say that something is more secure than before.

I also want to store some encrypted files on my cloud account, without the need to trust dropbox (or any cloud hosting file provider for that reason). As an extra security layer on top of dropbox, I use encfs and this blog post is a mini tutorial of a proof of concept.

EncFS - Encrypted Virtual Filesystem

(definition from encfs github account)

EncFS creates a virtual encrypted filesystem which stores encrypted data in the rootdir directory and makes the unencrypted data visible at the mountPoint directory. The user must supply a password which is used to (indirectly) encrypt both filenames and file contents.

That means that you can store your encrypted files somewhere and mount the decrypted files on folder on your computer.

Disclaimer: I dont know how secure is encfs. It is an extra layer that doesnt need any root access (except the installation part) for end users and it is really simple to use. There is a useful answer on stackexchange that you night like to read .

For more information on enfs you can also visit EncFS - Wikipedia Page

Install EncFS

  • archlinux

    $ sudo pacman -S --noconfirm encfs

  • fedora

    $ sudo dnf -y install fuse-encfs

  • ubuntu

    $ sudo apt-get install -y encfs

How does Encfs work ?

  • You have two(2) directories. The source and the mountpoint.
  • You encrypt and store the files in the source directory with a password.
  • You can view/edit your files in cleartext, in the mount point.
  1. Create a folder inside dropbox
    eg. /home/ebal/Dropbox/Boostnote

  2. Create a folder outside of dropbox
    eg. /home/ebal/Boostnote

both folders are complete empty.

  1. Choose a long password.
    just for testing, I am using a SHA256 message digest from an image that I can found on the internet!
    eg. sha256sum /home/ebal/secret.png

that means, I dont know the password but I can re-create it whenever I hash the image.

BE Careful This suggestion is an example - only for testing. The proper way is to use a random generated long password from your key password manager eg. KeePassX

How does dropbox works?

The dropbox-client is monitoring your /home/ebal/Dropbox/ directory for any changes so that can sync your files on your account.

You dont need dropbox running to use encfs.

Running the dropbox-client is the easiest way, but you can always use a sync client eg. rclone to sync your encrypted file to dropbox (or any cloud storage).

I guess it depends on your thread model. For this proof-of-concept article I run dropbox-client daemon in my background.


Create and Mount

Now is the time to mount the source directory inside dropbox with our mount point:

$ sha256sum /home/ebal/secret.png |
    awk '{print $1}' |
    encfs -S -s -f /home/ebal/Dropbox/Boostnote/ /home/ebal/Boostnote/

Reminder: EncFs works with absolute paths!

Check Mount Point

$ mount | egrep -i encfs
encfs on /home/ebal/Boostnote type fuse.encfs

View Files on Dropbox

Files inside dropbox:


View Files on the Mount Point


Unmount EncFS Mount Point

When you mount the source directory, encfs has an option to auto-umount the mount point on idle.
Or you can use the below command on demand:

$ fusermount -u /home/ebal/Boostnote

On another PC

The simplicity of this approach is when you want to access these files on another PC.
dropbox-client has already synced your encrypted files.
So the only thing you have to do, is to type on this new machine the exact same command as in Create & Mount chapter.

$ sha256sum /home/ebal/secret.png |
    awk '{print $1}' |
    encfs -S -s -f /home/ebal/Dropbox/Boostnote/ /home/ebal/Boostnote/


How about Android ?

You can use Cryptonite.

Cryptonite can use EncFS and TrueCrypt on Android and you can find the app on Google Play


Tag(s): encfs, dropbox