[this is a technical blog post, but easy to follow]
recently I had to setup and present my idea of a ssh bastion host. You may have already heard this as jump host or a security ssh hoping station or ssh gateway or even something else.
The main concept
Disclaimer: This is just a proof of concept (PoC). May need a few adjustments.
The destination VM may be on another VPC, perhaps it does not have a public DNS or even a public IP. Think of this VM as not accessible. Only the ssh bastion server can reach this VM. So we need to first reach the bastion.
To begin with, I will share my initial sshd_config to get an idea of my current ssh setup
AcceptEnv LANG LC_* ChallengeResponseAuthentication no Compression no MaxSessions 3 PasswordAuthentication no PermitRootLogin no Port 12345 PrintMotd no Subsystem sftp /usr/lib/openssh/sftp-server UseDNS no UsePAM yes X11Forwarding no AllowUsers ebal
- I only allow, user ebal to connect via ssh.
- I do not allow the root user to login via ssh.
- I use ssh keys instead of passwords.
This configuration is almost identical to both VMs
- bastion (the name of the VM that acts as a bastion server)
- VM (the name of the destination VM that is behind a DMZ/firewall)
I am using the ssh config file to have an easier user experience when using ssh
Host bastion Hostname 127.71.16.12 Port 12345 IdentityFile ~/.ssh/id_ed25519.vm Host vm Hostname 220.127.116.11 Port 23456 Host * User ebal ServerAliveInterval 300 ServerAliveCountMax 10 ConnectTimeout=60
Create a new user to test this
Let us create a new user for testing.
$ sudo groupadd ebal_test $ sudo useradd -g ebal_test -m ebal_test $ id ebal_test uid=1000(ebal_test) gid=1000(ebal_test) groups=1000(ebal_test)
Copy .ssh directory from current user (<== lazy sysadmin)
$ sudo cp -ravx /home/ebal/.ssh/ /home/ebal_test/ $ sudo chown -R ebal_test:ebal_test /home/ebal_test/.ssh $ sudo ls -la ~ebal_test/.ssh/ total 12 drwxr-x---. 2 ebal_test ebal_test 4096 Sep 20 2019 . drwx------. 3 ebal_test ebal_test 4096 Jun 23 15:56 .. -r--r-----. 1 ebal_test ebal_test 181 Sep 20 2019 authorized_keys $ sudo ls -ld ~ebal_test/.ssh/ drwxr-x---. 2 ebal_test ebal_test 4096 Sep 20 2019 /home/ebal_test/.ssh/
bastion sshd config
Edit the ssh daemon configuration file to append the below entries
AllowUsers ebal ebal_test Match User ebal_test AllowAgentForwarding no AllowTcpForwarding yes X11Forwarding no PermitTunnel no GatewayPorts no ForceCommand echo 'This account can only be used for ProxyJump (ssh -J)'
Don’t forget to restart sshd
systemctl restart sshd
As you have seen above, I now allow two (2) users to access the ssh daemon (AllowUsers). This can also work with AllowGroups
Let’s try to connect to this bastion VM
$ ssh bastion -l ebal_test uptime This account can only be used for ProxyJump (ssh -J)
$ ssh bastion -l ebal_test This account can only be used for ProxyJump (ssh -J) Connection to 127.71.16.12 closed.
We can not login into this machine.
Let’s try with our personal user
$ ssh bastion -l ebal uptime 18:49:14 up 3 days, 9:14, 0 users, load average: 0.00, 0.00, 0.00
Let’s try from windows (mobaxterm)
mobaxterm is putty on steroids! There is also a portable version, so there is no need of installation. You can just download and extract it.
Now it is time to test our access to the destination VM
$ ssh VM ssh: connect to host 18.104.22.168 port 23456: Connection refused
$ ssh -J ebal_test@bastion ebal@vm uptime 19:07:25 up 22:24, 2 users, load average: 0.00, 0.01, 0.00
$ ssh -J ebal_test@bastion ebal@vm Last login: Tue Jun 23 19:05:29 2020 from 22.214.171.124 ebal@vm:~$ ebal@vm:~$ exit logout
Using this command
ssh -J ebal_test@bastion ebal@vm
- is telling the ssh client command to use the ProxyJump feature.
- Using the user ebal_test on bastion machine and
- connect with the user ebal on vm.
So we can have different users!
Now, it is time to put everything under our
Host bastion Hostname 127.71.16.12 Port 12345 User ebal_test IdentityFile ~/.ssh/id_ed25519.vm Host vm Hostname 126.96.36.199 ProxyJump bastion User ebal Port 23456
and try again
$ ssh vm uptime 19:17:19 up 22:33, 1 user, load average: 0.22, 0.11, 0.03
mobaxterm with bastion
It is a known fact, that my favorite hosting provider is edis. I’ve seen them improving their services all these years, without forgeting their customers. Their support is great and I am really happy working with them.
That said, they dont offer (yet) a public infrastructre API like hetzner, linode or digitalocean but they offer an Auto Installer option to configure your VPS via a post-install shell script, put your ssh key and select your basic OS image.
I am experimenting with this option the last few weeks, but I wanted to use my currect cloud-init configuration file without making many changes. The goal is to produce a VPS image that when finished will be ready to accept my ansible roles without making any addition change or even login to this VPS.
So here is my current solution on how to use the post-install option to provide my current cloud-init configuration!
Josh Powers @ DebConf17
I will not get into cloud-init details in this blog post, but tldr; has stages, has modules, you provide your own user-data file (yaml) and it supports datasources. All these things is for telling cloud-init what to do, what to configure and when to configure it (in which step).
I am going to use NoCloud datastore for this experiment.
so I need to configure these two (2) files
My first entry in the post-install shell script should be
apt-get update && apt-get install cloud-init
thus I can be sure of two (2) things. First the VPS has already network access and I dont need to configure it, and second install cloud-init software, just to be sure that is there.
I try to keep my user-data file very simple but I would like to configure hostname and the sshd port.
Add a single user, provide a public ssh key for authentication and enable sudo access to this user.
users: - name: ebal ssh_import_id: - gh:ebal shell: /bin/bash sudo: ALL=(ALL) NOPASSWD:ALL
- Change sshd port
- Disable Root
- Disconnect Idle Sessions
- Disable Password Auth
- Disable X Forwarding
- Allow only User (or group)
write_files: - path: /etc/ssh/sshd_config content: | Port $SSHDPORT PermitRootLogin no ChallengeResponseAuthentication no ClientAliveInterval 600 ClientAliveCountMax 3 UsePAM yes UseDNS no X11Forwarding no PrintMotd no AcceptEnv LANG LC_* Subsystem sftp /usr/lib/openssh/sftp-server PasswordAuthentication no AllowUsers ebal
ufw allow $SSHDPORT/tcp && ufw enable
and last but not least, I need to remove cloud-init in the end
apt-get -y autoremove --purge cloud-init lxc lxd snapd
Post Install Shell script
let’s put everything together
#!/bin/bash apt-get update && apt-get install cloud-init HOSTNAME="complimentary" SSHDPORT=22422 mkdir -p /var/lib/cloud/seed/nocloud-net # Meta Data cat > /var/lib/cloud/seed/nocloud-net/meta-data <<EOF dsmode: local EOF # User Data cat > /var/lib/cloud/seed/nocloud-net/user-data <<EOF #cloud-config disable_root: true ssh_pwauth: no users: - name: ebal ssh_import_id: - gh:ebal shell: /bin/bash sudo: ALL=(ALL) NOPASSWD:ALL write_files: - path: /etc/ssh/sshd_config content: | Port $SSHDPORT PermitRootLogin no ChallengeResponseAuthentication no UsePAM yes UseDNS no X11Forwarding no PrintMotd no AcceptEnv LANG LC_* Subsystem sftp /usr/lib/openssh/sftp-server PasswordAuthentication no AllowUsers ebal # Set TimeZone timezone: Europe/Athens HOSTNAME: $HOSTNAME # Install packages packages: - figlet - mlocate - python3-apt - bash-completion # Update/Upgrade & Reboot if necessary package_update: true package_upgrade: true package_reboot_if_required: true # PostInstall runcmd: - figlet $HOSTNAME > /etc/motd - updatedb # Firewall - ufw allow $SSHDPORT/tcp && ufw enable # Remove cloud-init - apt-get -y autoremove --purge cloud-init lxc lxd snapd - apt-get -y --purge autoremove - apt -y autoclean - apt -y clean all EOF cloud-init clean --logs cloud-init init --local
That’s it !
After a while (needs a few reboot) our VPS is up & running and we can use ansible to configure it.
last days events, made me rethink of this story.
I am not the hero of the story.
I was in my early 20s, working part time on the tech lab of my uni. In this lab I met another student, I will call him Bob instead of his real name. I was just a couple months away to get my degree. He was ten years older than me, still trying to go through the studies to get his. We met and for the next couple of weeks, worked together, both part time in this lab. Bob was deaf. He could speak but due to the fact that he could not hear his voice, the words he made were not very clear. He was struggling with the courses. Bob was able to read lips but you had to speak directly to him and not very fast.
The majority of our courses had custom textbook and they were difficult. Dual courses, theory and lab was not always on the same subject. Theory was about compilers, lab was about pascal (just to get an idea). It was a difficult time for me. Back then (end of ’90s - begging of ’00s) the internet was not the place it is today. To solve a problem, you had to find the reference manual, read it through, understand it and learn from this process. Nowadays most of us are using search engines to copy/paste commands of software solutions. It wasn’t like that back then.
Bob was a good worker. But it was very difficult for him. He could pass some of the labs but he had problems with the theory courses. Ten years of his life in uni. I tried to help him with some of the courses. I was happy I was able to help, he was happy to find someone to help him. But I could not do much. To explain something back then it was not very easy for me. Also I had to slow my speech, find simple words and somehow translate some of the terminology to something that Bob could understand. I wasn’t the best person to do that and he knew it. I don’t know if I ever managed to actually helped him.
One time, I asked him:
- Why are you not attending the theory courses?
- I can read lips, but it is very difficult to understand if the professor is not speaking directly to me. Also when they turn their back to write something on the board, I can not see their lips. I spent more time guessing what they are trying to explain that the assumption I make is usually wrong. So I can not participate in those classes. I have to read alone from the textbook, buy additional material, search the internet and find someone to help me. We have so many courses that it takes two,three or even more times for me to pass a course. I am trying, but on my own time.
I had never understood the privilege I had, till that moment.
This was a true lesson for me. It was hard, It was difficult. I was in all theory classes, I was active, engaged, worked with other students together, asked gazillion questions and it was still difficult for me. You know what? It was 10times harder for people with a disability ! But I never, ever had any idea, I was looking everything from my perspective and this is not how you build a community or a society. You need empathy and understanding.
He also had some bitterness, it was not fair to him. Understandable but he also was angry with the system. With the systematic exclusion. He had a bad mouth for their peer students and our teachers. He had enough. Some times he wanted to break everything in the lab, instead of fixing it. From time to time, he was depressed, angry but a few times he was also happy. When he worked in the lab, he put his soul in his work but the majority of people didn’t expect much of him and he -sometimes- he would not even try much. This was his life. People saw him as a person with a disability, and treat him like that. He was also proud. Reaching a goal, passing a course, achieving something, overcoming his disability against all odds.
One day, I gather my entire courage to ask him, point blank a very stupid question:
- Why are you still here? Ten years of your life to get a degree? With no help from anyone? I do not think I could do it.
I will never forget his answer. Till this day, I am still thinking about what he said to me.
- When I use the computer, the person on the other side, does not know that I am deaf. They do not judge me by my disability, they actually don’t care at all. They speak to me, as if I am a person.
Lost every word and almost broke into tears.