Evaggelos Balaskas - System Engineer

[this is a technical blog post, but easy to follow]

recently I had to setup and present my idea of a ssh bastion host. You may have already heard this as jump host or a security ssh hoping station or ssh gateway or even something else.

The main concept

Disclaimer: This is just a proof of concept (PoC). May need a few adjustments.

The destination VM may be on another VPC, perhaps it does not have a public DNS or even a public IP. Think of this VM as not accessible. Only the ssh bastion server can reach this VM. So we need to first reach the bastion.

SSH Config

To begin with, I will share my initial sshd_config to get an idea of my current ssh setup

AcceptEnv LANG LC_*
ChallengeResponseAuthentication no
Compression no
MaxSessions 3
PasswordAuthentication no
PermitRootLogin no
Port 12345
PrintMotd no
Subsystem sftp  /usr/lib/openssh/sftp-server
UseDNS no
UsePAM yes
X11Forwarding no
AllowUsers ebal

• I only allow, user ebal to connect via ssh.
• I do not allow the root user to login via ssh.
• I use ssh keys instead of passwords.

This configuration is almost identical to both VMs

• bastion (the name of the VM that acts as a bastion server)
• VM (the name of the destination VM that is behind a DMZ/firewall)

~/.ssh/config

I am using the ssh config file to have an easier user experience when using ssh

Host bastion
     Hostname 127.71.16.12
     Port 12345
     IdentityFile ~/.ssh/id_ed25519.vm

Host vm
     Hostname 192.13.13.186
     Port 23456

Host *
    User ebal
    ServerAliveInterval 300
    ServerAliveCountMax 10
    ConnectTimeout=60

Create a new user to test this

Let us create a new user for testing.

User/Group

$ sudo groupadd ebal_test
$ sudo useradd -g ebal_test -m ebal_test

$ id ebal_test
uid=1000(ebal_test) gid=1000(ebal_test) groups=1000(ebal_test)

Perms

Copy .ssh directory from current user (<== lazy sysadmin)

$ sudo cp -ravx /home/ebal/.ssh/ /home/ebal_test/
$ sudo chown -R ebal_test:ebal_test /home/ebal_test/.ssh

$ sudo ls -la ~ebal_test/.ssh/
total 12
drwxr-x---. 2 ebal_test ebal_test 4096 Sep 20  2019 .
drwx------. 3 ebal_test ebal_test 4096 Jun 23 15:56 ..
-r--r-----. 1 ebal_test ebal_test  181 Sep 20  2019 authorized_keys

$ sudo ls -ld ~ebal_test/.ssh/
drwxr-x---. 2 ebal_test ebal_test 4096 Sep 20  2019 /home/ebal_test/.ssh/

bastion sshd config

Edit the ssh daemon configuration file to append the below entries

cat /etc/ssh/sshd_config

AllowUsers ebal ebal_test

Match User ebal_test
   AllowAgentForwarding no
   AllowTcpForwarding yes
   X11Forwarding no
   PermitTunnel no
   GatewayPorts no
   ForceCommand echo 'This account can only be used for ProxyJump (ssh -J)'

Don’t forget to restart sshd

systemctl restart sshd

As you have seen above, I now allow two (2) users to access the ssh daemon (AllowUsers). This can also work with AllowGroups

Testing bastion

Let’s try to connect to this bastion VM

$ ssh bastion -l ebal_test uptime
This account can only be used for ProxyJump (ssh -J)

$ ssh bastion -l ebal_test
This account can only be used for ProxyJump (ssh -J)
Connection to 127.71.16.12 closed.

Interesting …

We can not login into this machine.

Let’s try with our personal user

$ ssh bastion -l ebal uptime
 18:49:14 up 3 days,  9:14,  0 users,  load average: 0.00, 0.00, 0.00

Perfect.

Let’s try from windows (mobaxterm)

mobaxterm is putty on steroids! There is also a portable version, so there is no need of installation. You can just download and extract it.

Interesting…

Destination VM

Now it is time to test our access to the destination VM

$ ssh VM
ssh: connect to host 192.13.13.186 port 23456: Connection refused

bastion

$ ssh -J ebal_test@bastion ebal@vm uptime
 19:07:25 up 22:24,  2 users,  load average: 0.00, 0.01, 0.00

$ ssh -J ebal_test@bastion ebal@vm
Last login: Tue Jun 23 19:05:29 2020 from 94.242.59.170
ebal@vm:~$
ebal@vm:~$ exit
logout

Success !

Explain Command

Using this command

ssh -J ebal_test@bastion ebal@vm

• is telling the ssh client command to use the ProxyJump feature.
• Using the user ebal_test on bastion machine and
• connect with the user ebal on vm.

So we can have different users!

ssh/config

Now, it is time to put everything under our ~./ssh/config file

Host bastion
     Hostname 127.71.16.12
     Port 12345
     User ebal_test
     IdentityFile ~/.ssh/id_ed25519.vm

Host vm
     Hostname 192.13.13.186
     ProxyJump bastion
     User ebal
     Port 23456

and try again

$ ssh vm uptime
 19:17:19 up 22:33,  1 user,  load average: 0.22, 0.11, 0.03

mobaxterm with bastion

It is a known fact, that my favorite hosting provider is edis. I’ve seen them improving their services all these years, without forgeting their customers. Their support is great and I am really happy working with them.

That said, they dont offer (yet) a public infrastructre API like hetzner, linode or digitalocean but they offer an Auto Installer option to configure your VPS via a post-install shell script, put your ssh key and select your basic OS image.

I am experimenting with this option the last few weeks, but I wanted to use my currect cloud-init configuration file without making many changes. The goal is to produce a VPS image that when finished will be ready to accept my ansible roles without making any addition change or even login to this VPS.

So here is my current solution on how to use the post-install option to provide my current cloud-init configuration!

cloud-init

cloud-init documentation

Josh Powers @ DebConf17

I will not get into cloud-init details in this blog post, but tldr; has stages, has modules, you provide your own user-data file (yaml) and it supports datasources. All these things is for telling cloud-init what to do, what to configure and when to configure it (in which step).

NoCloud Seed

I am going to use NoCloud datastore for this experiment.

so I need to configure these two (2) files

/var/lib/cloud/seed/nocloud-net/meta-data
/var/lib/cloud/seed/nocloud-net/user-data

Install cloud-init

My first entry in the post-install shell script should be

apt-get update && apt-get install cloud-init

thus I can be sure of two (2) things. First the VPS has already network access and I dont need to configure it, and second install cloud-init software, just to be sure that is there.

Variables

I try to keep my user-data file very simple but I would like to configure hostname and the sshd port.

HOSTNAME="complimentary"
SSHDPORT=22422

Users

Add a single user, provide a public ssh key for authentication and enable sudo access to this user.

users:
  - name: ebal
    ssh_import_id:
      - gh:ebal
    shell: /bin/bash
    sudo: ALL=(ALL) NOPASSWD:ALL

Hardening SSH

• Change sshd port
• Disable Root
• Disconnect Idle Sessions
• Disable Password Auth
• Disable X Forwarding
• Allow only User (or group)

write_files:
  - path: /etc/ssh/sshd_config
    content: |
        Port $SSHDPORT
        PermitRootLogin no
        ChallengeResponseAuthentication no
        ClientAliveInterval 600
        ClientAliveCountMax 3
        UsePAM yes
        UseDNS no
        X11Forwarding no
        PrintMotd no
        AcceptEnv LANG LC_*
        Subsystem sftp  /usr/lib/openssh/sftp-server
        PasswordAuthentication no
        AllowUsers ebal

enable firewall

ufw allow $SSHDPORT/tcp && ufw enable

remove cloud-init

and last but not least, I need to remove cloud-init in the end

apt-get -y autoremove --purge cloud-init lxc lxd snapd

Post Install Shell script

let’s put everything together

#!/bin/bash

apt-get update && apt-get install cloud-init

HOSTNAME="complimentary"
SSHDPORT=22422

mkdir -p /var/lib/cloud/seed/nocloud-net

# Meta Data
cat > /var/lib/cloud/seed/nocloud-net/meta-data <<EOF
dsmode: local
EOF

# User Data
cat > /var/lib/cloud/seed/nocloud-net/user-data <<EOF
#cloud-config

disable_root: true
ssh_pwauth: no

users:
  - name: ebal
    ssh_import_id:
      - gh:ebal
    shell: /bin/bash
    sudo: ALL=(ALL) NOPASSWD:ALL

write_files:
  - path: /etc/ssh/sshd_config
    content: |
        Port $SSHDPORT
        PermitRootLogin no
        ChallengeResponseAuthentication no
        UsePAM yes
        UseDNS no
        X11Forwarding no
        PrintMotd no
        AcceptEnv LANG LC_*
        Subsystem sftp  /usr/lib/openssh/sftp-server
        PasswordAuthentication no
        AllowUsers ebal

# Set TimeZone
timezone: Europe/Athens

HOSTNAME: $HOSTNAME

# Install packages
packages:
  - figlet
  - mlocate
  - python3-apt
  - bash-completion

# Update/Upgrade & Reboot if necessary
package_update: true
package_upgrade: true
package_reboot_if_required: true

# PostInstall
runcmd:
  - figlet $HOSTNAME > /etc/motd
  - updatedb
  # Firewall
  - ufw allow $SSHDPORT/tcp && ufw enable
  # Remove cloud-init
  - apt-get -y autoremove --purge cloud-init lxc lxd snapd
  - apt-get -y --purge autoremove
  - apt -y autoclean
  - apt -y clean all

EOF

cloud-init clean --logs

cloud-init init --local

That’s it !

After a while (needs a few reboot) our VPS is up & running and we can use ansible to configure it.

last days events, made me rethink of this story.

I am not the hero of the story.

I was in my early 20s, working part time on the tech lab of my uni. In this lab I met another student, I will call him Bob instead of his real name. I was just a couple months away to get my degree. He was ten years older than me, still trying to go through the studies to get his. We met and for the next couple of weeks, worked together, both part time in this lab. Bob was deaf. He could speak but due to the fact that he could not hear his voice, the words he made were not very clear. He was struggling with the courses. Bob was able to read lips but you had to speak directly to him and not very fast.

The majority of our courses had custom textbook and they were difficult. Dual courses, theory and lab was not always on the same subject. Theory was about compilers, lab was about pascal (just to get an idea). It was a difficult time for me. Back then (end of ’90s - begging of ’00s) the internet was not the place it is today. To solve a problem, you had to find the reference manual, read it through, understand it and learn from this process. Nowadays most of us are using search engines to copy/paste commands of software solutions. It wasn’t like that back then.

Bob was a good worker. But it was very difficult for him. He could pass some of the labs but he had problems with the theory courses. Ten years of his life in uni. I tried to help him with some of the courses. I was happy I was able to help, he was happy to find someone to help him. But I could not do much. To explain something back then it was not very easy for me. Also I had to slow my speech, find simple words and somehow translate some of the terminology to something that Bob could understand. I wasn’t the best person to do that and he knew it. I don’t know if I ever managed to actually helped him.

One time, I asked him:

• Why are you not attending the theory courses?

he replied

• I can read lips, but it is very difficult to understand if the professor is not speaking directly to me. Also when they turn their back to write something on the board, I can not see their lips. I spent more time guessing what they are trying to explain that the assumption I make is usually wrong. So I can not participate in those classes. I have to read alone from the textbook, buy additional material, search the internet and find someone to help me. We have so many courses that it takes two,three or even more times for me to pass a course. I am trying, but on my own time.

I had never understood the privilege I had, till that moment.

This was a true lesson for me. It was hard, It was difficult. I was in all theory classes, I was active, engaged, worked with other students together, asked gazillion questions and it was still difficult for me. You know what? It was 10times harder for people with a disability ! But I never, ever had any idea, I was looking everything from my perspective and this is not how you build a community or a society. You need empathy and understanding.

He also had some bitterness, it was not fair to him. Understandable but he also was angry with the system. With the systematic exclusion. He had a bad mouth for their peer students and our teachers. He had enough. Some times he wanted to break everything in the lab, instead of fixing it. From time to time, he was depressed, angry but a few times he was also happy. When he worked in the lab, he put his soul in his work but the majority of people didn’t expect much of him and he -sometimes- he would not even try much. This was his life. People saw him as a person with a disability, and treat him like that. He was also proud. Reaching a goal, passing a course, achieving something, overcoming his disability against all odds.

One day, I gather my entire courage to ask him, point blank a very stupid question:

• Why are you still here? Ten years of your life to get a degree? With no help from anyone? I do not think I could do it.

I will never forget his answer. Till this day, I am still thinking about what he said to me.

• When I use the computer, the person on the other side, does not know that I am deaf. They do not judge me by my disability, they actually don’t care at all. They speak to me, as if I am a person.

Lost every word and almost broke into tears.