Evaggelos Balaskas - System Engineer

This blog post is written mostly as a reminder to myself!

Loop Assert Time Tests with Regex

test.yml

---
- hosts: localhost
  connection: local
  become: no
  gather_facts: no

  tasks:
  - name: Assert Time
    assert:
      that:
        - item | regex_findall('^(?:\d|[01]\d|2[0-3]):[0-5]\d(:[0-5]\d)?$')
      msg: "starTime should be HH:MM:SS or HH:MM"
    with_items:
      - "21:00"
      - "22:22"
      - "3:3"
      - "03:3"
      - "03:03"
      - "03:03:03"
      - "24:00"
      - "22:22:22"
      - "23:00:60"
      - "24:24:24"
      - "00:00:00"
      - "21:21"
      - "foo"
      - "foo:bar"
      - "True"
      - "False"
      - "1:01"
      - "01:01"
    ignore_errors: True

# vim: sts=2 sw=2 ts=2 et

output

PLAY [localhost] **************************************************************

TASK [Assert Time] ************************************************************
ok: [localhost] => (item=21:00) => {
    "changed": false,
    "item": "21:00",
    "msg": "All assertions passed"
}
ok: [localhost] => (item=22:22) => {
    "changed": false,
    "item": "22:22",
    "msg": "All assertions passed"
}
failed: [localhost] (item=3:3) => {
    "assertion": "item | regex_findall('^(?:\\d|[01]\\d|2[0-3]):[0-5]\\d(:[0-5]\\d)?$')",
    "changed": false,
    "evaluated_to": false,
    "item": "3:3",
    "msg": "starTime should be HH:MM:SS or HH:MM"
}
failed: [localhost] (item=03:3) => {
    "assertion": "item | regex_findall('^(?:\\d|[01]\\d|2[0-3]):[0-5]\\d(:[0-5]\\d)?$')",
    "changed": false,
    "evaluated_to": false,
    "item": "03:3",
    "msg": "starTime should be HH:MM:SS or HH:MM"
}
ok: [localhost] => (item=03:03) => {
    "changed": false,
    "item": "03:03",
    "msg": "All assertions passed"
}
ok: [localhost] => (item=03:03:03) => {
    "changed": false,
    "item": "03:03:03",
    "msg": "All assertions passed"
}
failed: [localhost] (item=24:00) => {
    "assertion": "item | regex_findall('^(?:\\d|[01]\\d|2[0-3]):[0-5]\\d(:[0-5]\\d)?$')",
    "changed": false,
    "evaluated_to": false,
    "item": "24:00",
    "msg": "starTime should be HH:MM:SS or HH:MM"
}
ok: [localhost] => (item=22:22:22) => {
    "changed": false,
    "item": "22:22:22",
    "msg": "All assertions passed"
}
failed: [localhost] (item=23:00:60) => {
    "assertion": "item | regex_findall('^(?:\\d|[01]\\d|2[0-3]):[0-5]\\d(:[0-5]\\d)?$')",
    "changed": false,
    "evaluated_to": false,
    "item": "23:00:60",
    "msg": "starTime should be HH:MM:SS or HH:MM"
}
failed: [localhost] (item=24:24:24) => {
    "assertion": "item | regex_findall('^(?:\\d|[01]\\d|2[0-3]):[0-5]\\d(:[0-5]\\d)?$')",
    "changed": false,
    "evaluated_to": false,
    "item": "24:24:24",
    "msg": "starTime should be HH:MM:SS or HH:MM"
}
ok: [localhost] => (item=00:00:00) => {
    "changed": false,
    "item": "00:00:00",
    "msg": "All assertions passed"
}
ok: [localhost] => (item=21:21) => {
    "changed": false,
    "item": "21:21",
    "msg": "All assertions passed"
}
failed: [localhost] (item=foo) => {
    "assertion": "item | regex_findall('^(?:\\d|[01]\\d|2[0-3]):[0-5]\\d(:[0-5]\\d)?$')",
    "changed": false,
    "evaluated_to": false,
    "item": "foo",
    "msg": "starTime should be HH:MM:SS or HH:MM"
}
failed: [localhost] (item=foo:bar) => {
    "assertion": "item | regex_findall('^(?:\\d|[01]\\d|2[0-3]):[0-5]\\d(:[0-5]\\d)?$')",
    "changed": false,
    "evaluated_to": false,
    "item": "foo:bar",
    "msg": "starTime should be HH:MM:SS or HH:MM"
}
failed: [localhost] (item=True) => {
    "assertion": "item | regex_findall('^(?:\\d|[01]\\d|2[0-3]):[0-5]\\d(:[0-5]\\d)?$')",
    "changed": false,
    "evaluated_to": false,
    "item": "True",
    "msg": "starTime should be HH:MM:SS or HH:MM"
}
failed: [localhost] (item=False) => {
    "assertion": "item | regex_findall('^(?:\\d|[01]\\d|2[0-3]):[0-5]\\d(:[0-5]\\d)?$')",
    "changed": false,
    "evaluated_to": false,
    "item": "False",
    "msg": "starTime should be HH:MM:SS or HH:MM"
}
ok: [localhost] => (item=1:01) => {
    "changed": false,
    "item": "1:01",
    "msg": "All assertions passed"
}
ok: [localhost] => (item=01:01) => {
    "changed": false,
    "item": "01:01",
    "msg": "All assertions passed"
}
...ignoring

PLAY RECAP ********************************************************************
localhost                  : ok=1    changed=0    unreachable=0    failed=0   

I was suspicious with a cron entry on a new ubuntu server cloud vm, so I ended up to be looking on the logs.

Authentication token is no longer valid; new one required

After a quick internet search,

# chage -l root

Last password change                                    : password must be changed
Password expires                                        : password must be changed
Password inactive                                       : password must be changed
Account expires                                         : never
Minimum number of days between password change          : 0
Maximum number of days between password change          : 99999
Number of days of warning before password expires       : 7

due to the password must be changed on the root account, the cron entry does not run as it should.

This ephemeral image does not need to have a persistent known password, as the notes suggest, and it doesn’t! Even so, we should change to root password when creating the VM.

Ansible

Ansible have a password plugin that we can use with lookup.

TLDR; here is the task:

- name: Generate Random Password
    user:
      name: root
      password: "{{ lookup('password','/dev/null encrypt=sha256_crypt length=32') }}"

after ansible-playbook runs

# chage -l root

Last password change                                    : Mar 10, 2019
Password expires                                        : never
Password inactive                                       : never
Account expires                                         : never
Minimum number of days between password change          : 0
Maximum number of days between password change          : 99999
Number of days of warning before password expires       : 7

and cron entry now runs as it should.

Password Plugin

Let explain how password plugin works.

Lookup needs at-least two (2) variables, the plugin name and a file to store the output. Instead, we will use /dev/null not to persist the password to a file.

To begin with it, a test ansible playbook:

- hosts: localhost
  gather_facts: False
  connection: local

  tasks:
    - debug:
        msg: "{{ lookup('password', '/dev/null') }}"
      with_sequence: count=5

Output:

ok: [localhost] => (item=1) => {
    "msg": "dQaVE0XwWti,7HMUgq::"
}
ok: [localhost] => (item=2) => {
    "msg": "aT3zqg.KjLwW89MrAApx"
}
ok: [localhost] => (item=3) => {
    "msg": "4LBNn:fVw5GhXDWh6TnJ"
}
ok: [localhost] => (item=4) => {
    "msg": "v273Hbox1rkQ3gx3Xi2G"
}
ok: [localhost] => (item=5) => {
    "msg": "NlwzHoLj8S.Y8oUhcMv,"
}

Length

In password plugin we can also use length variable:

msg: "{{ lookup('password', '/dev/null length=32') }}"

output:

ok: [localhost] => (item=1) => {
    "msg": "4.PEb6ycosnyL.SN7jinPM:AC9w2iN_q"
}
ok: [localhost] => (item=2) => {
    "msg": "s8L6ZU_Yzuu5yOk,ISM28npot4.KwQrE"
}
ok: [localhost] => (item=3) => {
    "msg": "L9QvLyNTvpB6oQmcF8WVFy.7jE4Q1K-W"
}
ok: [localhost] => (item=4) => {
    "msg": "6DMH8KqIL:kx0ngFe8:ri0lTK4hf,SWS"
}
ok: [localhost] => (item=5) => {
    "msg": "ByW11i_66K_0mFJVB37Mq2,.fBflepP9"
}

Characters

We can define a specific type of python string constants

• ascii_letters (ascii_lowercase and ascii_uppercase
• ascii_lowercase
• ascii_uppercase
• digits
• hexdigits
• letters (lowercase and uppercase)
• lowercase
• octdigits
• punctuation
• printable (digits, letters, punctuation and whitespace
• uppercase
• whitespace

eg.

msg: "{{ lookup('password', '/dev/null length=32 chars=ascii_lowercase') }}"

ok: [localhost] => (item=1) => {
    "msg": "vwogvnpemtdobjetgbintcizjjgdyinm"
}
ok: [localhost] => (item=2) => {
    "msg": "pjniysksnqlriqekqbstjihzgetyshmp"
}
ok: [localhost] => (item=3) => {
    "msg": "gmeoeqncdhllsguorownqbynbvdusvtw"
}
ok: [localhost] => (item=4) => {
    "msg": "xjluqbewjempjykoswypqlnvtywckrfx"
}
ok: [localhost] => (item=5) => {
    "msg": "pijnjfcpjoldfuxhmyopbmgdmgdulkai"
}

Encrypt

We can also define the encryption hash. Ansible uses passlib so the unix active encrypt hash algorithms are:

• passlib.hash.bcrypt - BCrypt
• passlib.hash.sha256_crypt - SHA-256 Crypt
• passlib.hash.sha512_crypt - SHA-512 Crypt

eg.

msg: "{{ lookup('password', '/dev/null length=32 chars=ascii_lowercase encrypt=sha512_crypt') }}"

ok: [localhost] => (item=1) => {
    "msg": "$6$BR96lZqN$jy.CRVTJaubOo6QISUJ9tQdYa6P6tdmgRi1/NQKPxwX9/Plp.7qETuHEhIBTZDTxuFqcNfZKtotW5q4H0BPeN."
}
ok: [localhost] => (item=2) => {
    "msg": "$6$ESf5xnWJ$cRyOuenCDovIp02W0eaBmmFpqLGGfz/K2jd1FOSVkY.Lsuo8Gz8oVGcEcDlUGWm5W/CIKzhS43xdm5pfWyCA4."
}
ok: [localhost] => (item=3) => {
    "msg": "$6$pS08v7j3$M4mMPkTjSwElhpY1bkVL727BuMdhyl4IdkGM7Mq10jRxtCSrNlT4cAU3wHRVxmS7ZwZI14UwhEB6LzfOL6pM4/"
}
ok: [localhost] => (item=4) => {
    "msg": "$6$m17q/zmR$JdogpVxY6MEV7nMKRot069YyYZN6g8GLqIbAE1cRLLkdDT3Qf.PImkgaZXDqm.2odmLN8R2ZMYEf0vzgt9PMP1"
}
ok: [localhost] => (item=5) => {
    "msg": "$6$tafW6KuH$XOBJ6b8ORGRmRXHB.pfMkue56S/9NWvrY26s..fTaMzcxTbR6uQW1yuv2Uy1DhlOzrEyfFwvCQEhaK6MrFFye."
}

Ansible is a wonderful software to automatically configure your systems. The default mode of using ansible is Push Model.

That means from your box, and only using ssh + python, you can configure your flee of machines.

Ansible is imperative. You define tasks in your playbooks, roles and they will run in a serial manner on the remote machines. The task will first check if needs to run and otherwise it will skip the action. And although we can use conditional to skip actions, tasks will perform all checks. For that reason ansible seems slow instead of other configuration tools. Ansible runs in serial mode the tasks but in psedo-parallel mode against the remote servers, to increase the speed. But sometimes you need to gather_facts and that would cost in execution time. There are solutions to cache the ansible facts in a redis (in memory key:value db) but even then, you need to find a work-around to speed your deployments.

But there is an another way, the Pull Mode!

Useful Reading Materials

to learn more on the subject, you can start reading these two articles on ansible-pull.

• Jan-Piet Mens

• Stavros Korokithakis

Pull Mode

So here how it looks:

You will first notice, that your ansible repository is moved from you local machine to an online git repository. For me, this is GitLab. As my git repo is private, I have created a Read-Only, time-limit, Deploy Token.

With that scenario, our (ephemeral - or not) VMs will pull their ansible configuration from the git repo and run the tasks locally. I usually build my infrastructure with Terraform by HashiCorp and make advance of cloud-init to initiate their initial configuration.

Cloud-init

The tail of my user-data.yml looks pretty much like this:

...
# Install packages
packages:
  - ansible

# Run ansible-pull
runcmd:
  - ansible-pull -U https://gitlab+deploy-token-XXXXX:YYYYYYYY@gitlab.com/username/myrepo.git 

Playbook

You can either create a playbook named with the hostname of the remote server, eg. node1.yml or use the local.yml as the default playbook name.

Here is an example that will also put ansible-pull into a cron entry. This is very useful because it will check for any changes in the git repo every 15 minutes and run ansible again.

- hosts: localhost

  tasks:
    - name: Ensure ansible-pull is running every 15 minutes
      cron:
        name: "ansible-pull"
        minute: "15"
        job: "ansible-pull -U https://gitlab+deploy-token-XXXXX:YYYYYYYY@gitlab.com/username/myrepo.git &> /dev/null"

    - name: Create a custom local vimrc file
      lineinfile:
        path: /etc/vim/vimrc.local
        line: 'set modeline'
        create: yes

    - name: Remove "cloud-init" package
      apt:
        name: "cloud-init"
        purge: yes
        state: absent

    - name: Remove useless packages from the cache
      apt:
        autoclean: yes

    - name: Remove dependencies that are no longer required
      apt:
        autoremove: yes

# vim: sts=2 sw=2 ts=2 et

In this short blog post, I will share a few ansible tips:

Print Ansible Variables

ansible -m debug -a 'var=vars' 127.0.0.1 | sed "1 s/^.*$/{/" | jq .

Ansible Conditionals

Variable contains string

matching string in a variable

  vars:
      var1: "hello world"

  - debug:
        msg: " {{ var1 }} "
    when:  ........

old way

  when: var1.find("hello") != -1

deprecated way

  when: var1 | search("hello")

Correct way

  when: var1 is search("hello")

Multiple Conditionals

Logical And

      when:
        - ansible_distribution == "Archlinux"
        - ansible_hostname is search("myhome")

Numeric Conditionals

getting variable from command line (or from somewhere else)

   -  set_fact:
       my_variable_keepday: "{{ keepday | default(7)  | int }}"

    - name: something
      debug:
        msg: "keepday : {{  my_variable_keepday }}"
      when:
        - my_variable_keepday|int >= 0
        - my_variable_keepday|int <= 10

Validate Time variable with ansible

I need to validate a variable. It should be ‘%H:%M:%S’
My work-around is to convert it to datetime so that I can validate it:

  tasks:
    - debug:
        msg: "{{ startTime | to_datetime('%H:%M:%S') }}"

First example: 21:30:15

True: "msg": "1900-01-01 21:30:15"

Second example: ‘25:00:00′

could not be converted to an dict.
The error was: time data '25:00:00' does not match format '%H:%M:%S'

This article will show how to install Arch Linux in Windows 10 under Windows Subsystem for Linux.

WSL

Prerequisite is to have enabled WSL on your Win10 and already reboot your machine.

You can enable WSL :

• Windows Settings
• Apps
• Apps & features
• Related settings -> Programs and Features (bottom)
• Turn Windows features on or off (left)

Store

After rebooting your Win10, you can use Microsoft Store to install a Linux distribution like Ubuntu. Archlinux is not an official supported linux distribution thus this guide !

Launcher

The easiest way to install Archlinux (or any Linux distro) is to download the wsldl from github. This project provides a generic Launcher.exe and any rootfs as source base. First thing is to rename Launcher.exe to Archlinux.exe.

ebal@myworklaptop:~$ mkdir -pv Archlinux
mkdir: created directory 'Archlinux'

ebal@myworklaptop:~$ cd Archlinux/

ebal@myworklaptop:~/Archlinux$ curl -sL -o Archlinux.exe https://github.com/yuk7/wsldl/releases/download/18122700/Launcher.exe
ebal@myworklaptop:~/Archlinux$ ls -l
total 320
-rw-rw-rw- 1 ebal ebal 143147 Feb 21 20:40 Archlinux.exe

RootFS

Next step is to download the latest archlinux root filesystem and create a new rootfs.tar.gz archive file, as wsldl uses this type.

ebal@myworklaptop:~/Archlinux$ curl -sLO http://ftp.otenet.gr/linux/archlinux/iso/latest/archlinux-bootstrap-2019.02.01-x86_64.tar.gz

ebal@myworklaptop:~/Archlinux$ ls -l
total 147392
-rw-rw-rw- 1 ebal ebal    143147 Feb 21 20:40 Archlinux.exe
-rw-rw-rw- 1 ebal ebal 149030552 Feb 21 20:42 archlinux-bootstrap-2019.02.01-x86_64.tar.gz

ebal@myworklaptop:~/Archlinux$ sudo tar xf archlinux-bootstrap-2019.02.01-x86_64.tar.gz

ebal@myworklaptop:~/Archlinux$  cd root.x86_64/

ebal@myworklaptop:~/Archlinux/root.x86_64$ ls
README  bin  boot  dev  etc  home  lib  lib64  mnt  opt  proc  root  run  sbin  srv  sys  tmp  usr  var

ebal@myworklaptop:~/Archlinux/root.x86_64$  sudo tar czf rootfs.tar.gz .
tar: .: file changed as we read it

ebal@myworklaptop:~/Archlinux/root.x86_64$ ls
README  bin  boot  dev  etc  home  lib  lib64  mnt  opt  proc  root  rootfs.tar.gz  run  sbin  srv  sys  tmp  usr  var

ebal@myworklaptop:~/Archlinux/root.x86_64$ du -sh rootfs.tar.gz
144M    rootfs.tar.gz

ebal@myworklaptop:~/Archlinux/root.x86_64$ sudo mv rootfs.tar.gz ../

ebal@myworklaptop:~/Archlinux/root.x86_64$ cd ..
ebal@myworklaptop:~/Archlinux$ ls
Archlinux.exe  archlinux-bootstrap-2019.02.01-x86_64.tar.gz  root.x86_64  rootfs.tar.gz

ebal@myworklaptop:~/Archlinux$
ebal@myworklaptop:~/Archlinux$ ls
Archlinux.exe  rootfs.tar.gz

ebal@myworklaptop:~$ mv Archlinux/ /mnt/c/Users/EvaggelosBalaskas/Downloads/ArchlinuxWSL
ebal@myworklaptop:~$

As you can see, I do a little clean up and I move the directory under windows filesystem.

Install & Verify

Microsoft Windows [Version 10.0.17134.619]
(c) 2018 Microsoft Corporation. All rights reserved.

C:UsersEvaggelosBalaskas>cd Downloads/ArchlinuxWSL

C:UsersEvaggelosBalaskasDownloadsArchlinuxWSL>dir
 Volume in drive C is Windows
 Volume Serial Number is 6C02-EE43

 Directory of C:UsersEvaggelosBalaskasDownloadsArchlinuxWSL

21-Feb-19  21:04    <DIR>          .
21-Feb-19  21:04    <DIR>          ..
21-Feb-19  20:40           143,147 Archlinux.exe
21-Feb-19  20:52       150,178,551 rootfs.tar.gz
               2 File(s)    150,321,698 bytes
               2 Dir(s)  374,579,486,720 bytes free

C:UsersEvaggelosBalaskasDownloadsArchlinuxWSL>Archlinux.exe
Installing...
Installation Complete!
Press any key to continue...

C:UsersEvaggelosBalaskasDownloadsArchlinuxWSL>Archlinux.exe run uname -a
Linux myworklaptop 4.4.0-17134-Microsoft #523-Microsoft Mon Dec 31 17:49:00 PST 2018 x86_64 GNU/Linux

C:UsersEvaggelosBalaskasDownloadsArchlinuxWSL>Archlinux.exe run cat /etc/os-release
NAME="Arch Linux"
PRETTY_NAME="Arch Linux"
ID=arch
BUILD_ID=rolling
ANSI_COLOR="0;36"
HOME_URL="https://www.archlinux.org/"
DOCUMENTATION_URL="https://wiki.archlinux.org/"
SUPPORT_URL="https://bbs.archlinux.org/"
BUG_REPORT_URL="https://bugs.archlinux.org/"

C:UsersEvaggelosBalaskasDownloadsArchlinuxWSL>Archlinux.exe run bash
[root@myworklaptop ArchlinuxWSL]#
[root@myworklaptop ArchlinuxWSL]# exit

Archlinux

C:UsersEvaggelosBalaskasDownloadsArchlinuxWSL>Archlinux.exe run bash
[root@myworklaptop ArchlinuxWSL]#

[root@myworklaptop ArchlinuxWSL]# date
Thu Feb 21 21:41:41 STD 2019

Remember, archlinux by default does not have any configuration. So you need to configure this instance !

Here are some basic configuration:

[root@myworklaptop ArchlinuxWSL]# echo nameserver 8.8.8.8 > /etc/resolv.conf

[root@myworklaptop ArchlinuxWSL]# cat > /etc/pacman.d/mirrorlist <<EOF
Server = http://ftp.otenet.gr/linux/archlinux/$repo/os/$arch
EOF

[root@myworklaptop ArchlinuxWSL]#  pacman-key --init

[root@myworklaptop ArchlinuxWSL]#  pacman-key --populate

[root@myworklaptop ArchlinuxWSL]# pacman -Syy

you are pretty much ready to use archlinux inside your windows 10 !!

Remove

You can remove Archlinux by simple:

 Archlinux.exe clean 

Default User

There is a simple way to use Archlinux within Windows Subsystem for Linux , by connecting with a default user.

But before configure ArchWSL, we need to create this user inside the archlinux instance:

[root@myworklaptop ArchWSL]# useradd -g 374 -u 374 ebal

[root@myworklaptop ArchWSL]# id ebal
uid=374(ebal) gid=374(ebal) groups=374(ebal)

[root@myworklaptop ArchWSL]# cp -rav /etc/skel/ /home/ebal
'/etc/skel/' -> '/home/ebal'
'/etc/skel/.bashrc' -> '/home/ebal/.bashrc'
'/etc/skel/.bash_profile' -> '/home/ebal/.bash_profile'
'/etc/skel/.bash_logout' -> '/home/ebal/.bash_logout'

chown -R ebal:ebal /home/ebal/

then exit the linux app and run:

> Archlinux.exe config --default-user ebal

and try to login again:

> Archlinux.exe run bash
[ebal@myworklaptop ArchWSL]$ 

[ebal@myworklaptop ArchWSL]$ cd ~

ebal@myworklaptop ~$ pwd -P
/home/ebal

Flatpak is a software utility for software deployment, package management, and application virtualization for Linux desktop computers. It provides a sandbox environment in which users can run applications in isolation from the rest of the system.

… in a nutshell, it is an isolate software bundle package which you can run with restricted permissions!

User Vs System

We can install flatpak applications for system-wide or for single-user. The last part does not need administrative access or any special permissions.

To use flatpak as a user, we have to add --user next to every flatpak command.

Applications

A flatpak application has a manifest that describes dependancies & permissions.

Repositories

A repository contains a list of application manifests & the flatpak package (code).

Branches

One of the best features of flatpak, is that we can have multile versions of a specific application. This is being done by using a different branch or version (like git). Most common branch is default.

Add flathub

To use/install a flatpak application, we have first to add a remote flatpack repository localy.
The most well-known flatpak repository is called: flathub

flatpak remote-add --user flathub https://dl.flathub.org/repo/flathub.flatpakrepo

Search Applications

$ flatpak search --user Signal

Description                                                                                     Application                                 Version              Branch              Remotes
Signal - Private messenger for the desktop                                                      org.signal.Signal                           1.20.0               stable              flathub

Install package as user

The default syntax of install packages is:

repository - application

$ flatpak install --user -y flathub com.dropbox.Client
$ flatpak install --user -y flathub org.signal.Signal

List

See how many packages to you have installed

$ flatpak list

Application
___________

com.dropbox.Client
com.jetbrains.PyCharm-Community
com.slack.Slack
com.visualstudio.code.oss
io.atom.Atom
org.signal.Signal

Run

To execute a flatpak application :

flatpak run com.slack.Slack
flatpak run org.signal.Signal

Update

and finally how to update your flatpack packages:

flatpak --user -y update

TLDR; Exit status value does not change when using redirection.

~> false
~> echo $?
1

~> true
~> echo $?
0

~> false > /dev/null
~> echo $?
1

~> true > /dev/null
~> echo $?
0

~> false 1> /dev/null
~> echo $?
1

~> true 1> /dev/null
~> echo $?
0

~> false 2> /dev/null
~> echo $?
1

~> true 2> /dev/null
~> echo $?
0

~> false &> /dev/null
~> echo $?
1

~> true &> /dev/null
~> echo $?
0

~> false 2>&1 >/dev/null
~> echo $?
1

~> true 2>&1 /dev/null
~> echo $?
0

~> false < /dev/null > /dev/null
~> echo $?
1

~> true < /dev/null > /dev/null
~> echo $?
0

Using Terraform by HashiCorp and cloud-init on Hetzner cloud provider.

Nowadays with the help of modern tools, we use our infrastructure as code. This approach is very useful because we can have Immutable design with our infra by declaring the state would like our infra to be. This also provide us with flexibility and a more generic way on how to handle our infra as lego bricks, especially on scaling.

UPDATE: 2019.01.22

Hetzner

We need to create an Access API Token within a new project under the console of hetzner cloud.

Copy this token and with that in place we can continue with terraform.
For the purposes of this article, I am going to use as the API token: 01234567890

Install Terraform

the latest terraform version at the time of writing this blog post is: v.11.11

$ curl -sL https://releases.hashicorp.com/terraform/0.11.11/terraform_0.11.11_linux_amd64.zip |
   bsdtar -xf- && chmod +x terraform
$ sudo mv terraform /usr/local/bin/

and verify it

$ terraform version

Terraform v0.11.11

Terraform Provider for Hetzner Cloud

To use the hetzner cloud via terraform, we need the terraform-provider-hcloud plugin.

hcloud, is part of terraform providers repository. So the first time of initialize our project, terraform will download this plugin locally.

Initializing provider plugins...
- Checking for available provider plugins on https://releases.hashicorp.com...
- Downloading plugin for provider "hcloud" (1.7.0)...
...
* provider.hcloud: version = "~> 1.7"

Compile hcloud

If you like, you can always build hcloud from the source code.
There are notes on how to build the plugin here Terraform Hetzner Cloud provider.

GitLab CI

or you can even download the artifact from my gitlab-ci repo.

Plugin directory

You will find the terraform hcloud plugin under your current directory:

./.terraform/plugins/linux_amd64/terraform-provider-hcloud_v1.7.0_x4

I prefer to copy the tf plugins centralized under my home directory:

$ mkdir -pv ~/.terraform/plugins/linux_amd64/
$ mv ./.terraform/plugins/linux_amd64/terraform-provider-hcloud_v1.7.0_x4 ~/.terraform.d/plugins/linux_amd64/terraform-provider-hcloud

or if you choose the artifact from gitlab:

$ curl -sL -o ~/.terraform/plugins/linux_amd64/terraform-provider-hcloud https://gitlab.com/ebal/terraform-provider-hcloud-ci/-/jobs/artifacts/master/raw/bin/terraform-provider-hcloud?job=run-build

That said, when working with multiple terraform projects you may be in a position that you need different versions of the same tf-plugin. In that case it is better to have them under your current working directory/project instead of your home directory. Perhaps one project needs v1.2.3 and another v4.5.6 of the same tf-plugin.

Hetzner Cloud API

Here is a few examples on how to use the Hetzner Cloud API:

$ export -p API_TOKEN="01234567890"

$ curl -sH "Authorization: Bearer $API_TOKEN" https://api.hetzner.cloud/v1/datacenters | jq -r .datacenters[].name
fsn1-dc8
nbg1-dc3
hel1-dc2
fsn1-dc14

$ curl -sH "Authorization: Bearer $API_TOKEN" https://api.hetzner.cloud/v1/locations | jq -r .locations[].name
fsn1
nbg1
hel1

$ curl -sH "Authorization: Bearer $API_TOKEN" https://api.hetzner.cloud/v1/images | jq -r .images[].name
ubuntu-16.04
debian-9
centos-7
fedora-27
ubuntu-18.04
fedora-28

hetzner.tf

At this point, we are ready to write our terraform file.
It can be as simple as this (CentOS 7):

# Set the variable value in *.tfvars file
# or using -var="hcloud_token=..." CLI option
variable "hcloud_token" {}

# Configure the Hetzner Cloud Provider
provider "hcloud" {
  token = "${var.hcloud_token}"
}

# Create a new server running centos
resource "hcloud_server" "node1" {
  name = "node1"
  image = "centos-7"
  server_type = "cx11"
}

Project_Ebal

or a more complex config: Ubuntu 18.04 LTS

# Project_Ebal
variable "hcloud_token" {}

# Configure the Hetzner Cloud Provider
provider "hcloud" {
  token = "${var.hcloud_token}"
}

# Create a new server running centos
resource "hcloud_server" "Project_Ebal" {
  name = "ebal_project"
  image = "ubuntu-18.04"
  server_type = "cx11"
  location = "nbg1"
}

Repository Structure

Although in this blog post we have a small and simple example of using hetzner cloud with terraform, on larger projects is usually best to have separated terraform files for variables, code and output. For more info, you can take a look here: VCS Repository Structure - Workspaces

  ├── variables.tf
  ├── main.tf
  ├── outputs.tf

Cloud-init

To use cloud-init with hetzner is very simple.
We just need to add this declaration user_data = "${file("user-data.yml")}" to terraform file.
So our previous tf is now this:

# Project_Ebal
variable "hcloud_token" {}

# Configure the Hetzner Cloud Provider
provider "hcloud" {
  token = "${var.hcloud_token}"
}

# Create a new server running centos
resource "hcloud_server" "Project_Ebal" {
  name = "ebal_project"
  image = "ubuntu-18.04"
  server_type = "cx11"
  location = "nbg1"
  user_data = "${file("user-data.yml")}"
}

to get the IP_Address of the virtual machine, I would also like to have an output declaration:

output "ipv4_address" {
  value = "${hcloud_server.ebal_project.ipv4_address}"
}

Clout-init

You will find more notes on cloud-init on a previous blog post: Cloud-init with CentOS 7.

below is an example of user-data.yml

#cloud-config

disable_root: true
ssh_pwauth: no

users:
  - name: ubuntu
    ssh_import_id:
     - gh:ebal
    shell: /bin/bash
    sudo: ALL=(ALL) NOPASSWD:ALL

# Set TimeZone
timezone: Europe/Athens

# Install packages
packages:
  - mlocate
  - vim
  - figlet

# Update/Upgrade & Reboot if necessary
package_update: true
package_upgrade: true
package_reboot_if_required: true

# Remove cloud-init
runcmd:
  - figlet Project_Ebal > /etc/motd
  - updatedb

Terraform

First thing with terraform is to initialize our environment.

Init

$ terraform init

Initializing provider plugins...

Terraform has been successfully initialized!

You may now begin working with Terraform. Try running "terraform plan" to see
any changes that are required for your infrastructure. All Terraform commands
should now work.

If you ever set or change modules or backend configuration for Terraform,
rerun this command to reinitialize your working directory. If you forget, other
commands will detect it and remind you to do so if necessary.

Plan

Of course it is not necessary to plan and then plan with out.
You can skip this step, here exist only for documentation purposes.

$ terraform plan

Refreshing Terraform state in-memory prior to plan...
The refreshed state will be used to calculate this plan, but will not be
persisted to local or remote state storage.

------------------------------------------------------------------------

An execution plan has been generated and is shown below.
Resource actions are indicated with the following symbols:
  + create

Terraform will perform the following actions:

  + hcloud_server.ebal_project
      id:            <computed>
      backup_window: <computed>
      backups:       "false"
      datacenter:    <computed>
      image:         "ubuntu-18.04"
      ipv4_address:  <computed>
      ipv6_address:  <computed>
      ipv6_network:  <computed>
      keep_disk:     "false"
      location:      "nbg1"
      name:          "ebal_project"
      server_type:   "cx11"
      status:        <computed>
      user_data:     "sk6134s+ys+wVdGITc+zWhbONYw="

Plan: 1 to add, 0 to change, 0 to destroy.

------------------------------------------------------------------------

Note: You didn't specify an "-out" parameter to save this plan, so Terraform
can't guarantee that exactly these actions will be performed if
"terraform apply" is subsequently run.

Out

$ terraform plan -out terraform.tfplan

Refreshing Terraform state in-memory prior to plan...
The refreshed state will be used to calculate this plan, but will not be
persisted to local or remote state storage.

------------------------------------------------------------------------

An execution plan has been generated and is shown below.
Resource actions are indicated with the following symbols:
  + create

Terraform will perform the following actions:

  + hcloud_server.ebal_project
      id:            <computed>
      backup_window: <computed>
      backups:       "false"
      datacenter:    <computed>
      image:         "ubuntu-18.04"
      ipv4_address:  <computed>
      ipv6_address:  <computed>
      ipv6_network:  <computed>
      keep_disk:     "false"
      location:      "nbg1"
      name:          "ebal_project"
      server_type:   "cx11"
      status:        <computed>
      user_data:     "sk6134s+ys+wVdGITc+zWhbONYw="

Plan: 1 to add, 0 to change, 0 to destroy.

------------------------------------------------------------------------

This plan was saved to: terraform.tfplan

To perform exactly these actions, run the following command to apply:
    terraform apply "terraform.tfplan"

Apply

$ terraform apply "terraform.tfplan"

hcloud_server.ebal_project: Creating...
  backup_window: "" => "<computed>"
  backups:       "" => "false"
  datacenter:    "" => "<computed>"
  image:         "" => "ubuntu-18.04"
  ipv4_address:  "" => "<computed>"
  ipv6_address:  "" => "<computed>"
  ipv6_network:  "" => "<computed>"
  keep_disk:     "" => "false"
  location:      "" => "nbg1"
  name:          "" => "ebal_project"
  server_type:   "" => "cx11"
  status:        "" => "<computed>"
  user_data:     "" => "sk6134s+ys+wVdGITc+zWhbONYw="
hcloud_server.ebal_project: Still creating... (10s elapsed)
hcloud_server.ebal_project: Still creating... (20s elapsed)
hcloud_server.ebal_project: Creation complete after 23s (ID: 1676988)

Apply complete! Resources: 1 added, 0 changed, 0 destroyed.

Outputs:

ipv4_address = 1.2.3.4

SSH and verify cloud-init

$ ssh 1.2.3.4 -l ubuntu

Welcome to Ubuntu 18.04.1 LTS (GNU/Linux 4.15.0-43-generic x86_64)

 * Documentation:  https://help.ubuntu.com
 * Management:     https://landscape.canonical.com
 * Support:        https://ubuntu.com/advantage

  System information as of Fri Jan 18 12:17:14 EET 2019

  System load:  0.41              Processes:           89
  Usage of /:   9.7% of 18.72GB   Users logged in:     0
  Memory usage: 8%                IP address for eth0: 1.2.3.4
  Swap usage:   0%

0 packages can be updated.
0 updates are security updates.

Destroy

Be Careful without providing a specific terraform out plan, terraform will destroy every tfplan within your working directory/project. So it is always a good practice to explicit destroy a specify resource/tfplan.

$ terraform destroy should better be:

$ terraform destroy -out terraform.tfplan

hcloud_server.ebal_project: Refreshing state... (ID: 1676988)

An execution plan has been generated and is shown below.
Resource actions are indicated with the following symbols:
  - destroy

Terraform will perform the following actions:

  - hcloud_server.ebal_project

Plan: 0 to add, 0 to change, 1 to destroy.

Do you really want to destroy all resources?
  Terraform will destroy all your managed infrastructure, as shown above.
  There is no undo. Only 'yes' will be accepted to confirm.

  Enter a value: yes

hcloud_server.ebal_project: Destroying... (ID: 1676988)
hcloud_server.ebal_project: Destruction complete after 1s

Destroy complete! Resources: 1 destroyed.

That’s it !