This is a continuation of my other post
I now have homeassistant, immich, and authentik docker containers exposed to the open internet. Homeassistant has built in 2FA and authentik is being used as the authentication for immich which supports 2FA. I went ahead and blocked connections from every country except for my own via cloudlfare (I’m aware this does almost nothing but I feel better about it).
At the moment, if my machine became compromised, I wouldn’t know. How do I monitor these docker containers? What’s a good way to block IPs based on failed login attempts? Is there a tool that could alert me if my machine was compromised? Any recommendations?
EDIT: Oh, and if you have any recommendations for settings I should change in the cloudflare dashboard, that would be great too; there’s a ton of options in there and a lot of them are defaulted to “off”
I’ve tried different approaches with fail2ban, crowdsec, VPNs, etc. What I settled on is to divide the data of my services in two categories: confidential and “I can live with it leaking”.
The ones that host confidential data is behind a VPN and has some basic monitoring on them.
The ones that are out in the public are behind a WAF from cloudflare with pretty restrictive rules.
Yes, cloudflare suck etc., but the value of stopping potential attacks before they reach your services is hard to match.
Just keep in mind: you need layers of different security measures to protect your services (such as backups, control of network traffic, monitoring and detection, and so on).
has some basic monitoring on them.
What monitoring software are you using?
I feel like the other measures you talked about (backups, condom of network traffic, etc) I’m doing ok on. Its really just the monitoring where I’m stuck. There’s so many options
Wazuh
Active responses are like fail2ban but better
I’ll look into it, thank you
By not making them publicly accessible. With Wireguard there’s really no reason.
Setup service to be active on a subnet, enable Wireguard to VPN into the subnet and use the services.
Yeah, I’m not gonna tell the 50 users of my plex server to set up wireguard on their devices so they can request movies and TV series on my overseer, when I can instead just use NPM to make it publically accessible with a password prompt
With Wireguard there’s really no reason.
Well, that’s kinda of a personal choice. If somebody needs to have services accessible by someone else besides him, that service can’t be behind a VPN (let’s face the truth: we know that we can’t ask all out relatives and friends to use a VPN).
If somebody needs to have services accessible by someone else besides him, that service can’t be behind a VPN
Again, this is the reason VPS’ exist. If that person needs access, then setup Wireguard…
It’s like saying you don’t need a front gate with an access code because then you would have to give out your own access code. But I mean, the lock has the ability to setup more access codes. And you’re saying the only viable option is the leave the gate open and hire a guard to manage access. It’s just… Weird and wrong.
Again, this is the reason VPS’ exist.
What? What’s the difference between a VPS and your home server? You may say that’s a good practice to separate things, so maybe have a a VM with public facing services and another with more private stuff reachable only with a VPN. But for something like Nextcloud, it needs to be public (if you’re not the only one using it), but it contains personal stuff and then comes the OP request!
There’s also something to be said about some services being cordoned off in a VPN while leaving some public with security. I don’t necessarily want everyone within my full network if all I want is to share one service with them.
For that, you can restrict access to a single service with iptables.
I agree with WG however I need https for a few locally hosted items like actual budget so I have that through nginx proxy manager. I was debating adding Authelia in front with some of my others (audiobook shelf, home assistant and music assistant) as sometimes I disconnect from my home network and forget to reconnect.
Why not swap from nginx-proxy-manager to Caddy2, which can handle everything? SSL and reverse_proxy?
Just out of curiosity, why do you disconnect from your home VPN?
There should be an option in your phone VPN setup to reconnect if app X is being used.
Check out crowdsec. Like fail2ban, but with crowdsourced lists on top.
will do, thanks
I would put this stuff behind VPN.
So there is https://en.wikipedia.org/wiki/Fail2ban which helps already to some degree.
But what are you trying to prevent? You have your services in a docker container, hopefully not running as root, which already makes it difficult to break out even if through a bug someone would be able to get access to the docker container.
I mean its not like your stuff is very important for someone to break in like the pentagon, you probably just have some photos from your phone on it, some lights can be switched on and off and some temperatures read.
I’m not trying to say that you should not care about it but I’m trying to figure out what your threat model is.
I feel weird about having those apps on the internet and basically being blind to threats. I mean yeah, I’m not a target on anyone’s list and most IPs visiting the site are bots but I would still like to know what’s going on.
I don’t work in tech for a living, this is just a hobby for me so I have limited time to work on this stuff and do research. It’s very possible I fucked something up and don’t know it. I figured if I at least got an alert that said “hey, your immich server db was dumped and sent to <insert IP>”, I could at least turn it off
Yah, it’s just a hobby for you, but it’s also a hobby for script kiddies to use Shodan to find people with out of date web interfaces and pop them. I tell you right now, the Immich team would be the first to say not to put their application publicly accessible.
Just don’t get into this practice, it ends in tears and is way more maintenance to stay protected than just setting up tailscale and using that.
By the time you get the alert and act on it, it’s too late.
Don’t expose these things to the open internet; VPN back into your network and access them.
IOT botnets are a thing. And if someone wanted to fire sell the US, all the vulnerable home networks would be on the table too. Great for a bit if extra chaos.
Auth portal for VPN tunnell -> Authelia -> fail2ban -> VLAN with services only.
ELK stack monitors the LAN. (Including VLAN)
Keep that VLAN segmented. You’re good unless you’re a DOGE employee, then I’d recommend quite a bit more security.
This is the way. Layer 3 separation for services you wish to access outside of the home network and the rest of your stuff, with a VPN endpoint exposed for remote access.
It may be overkill, but I have several VLANs for specific traffic:
- DMZ - for Wireguard (and if I ever want to stand up a Honeypot)
- Services - *arr stack, some Kubes things for remote development
- IoT - any smart things like thermostat, home assistant, etc
- Trusted - primary at home network for laptops, HTPCs, etc
There are two new additions: a ext-vpn VLAN and a egress-vpn VLAN. I spun up a VM that’s dual homed running its own Wireguard/OpenVPN client on the egress side, serving DHCP on the ext-vpn side. The latter has its own wireless ssid so that anyone who connects to it is automatically on a VPN into a non-US country.
I’ve seen a bunch of people recommend Authelia. Do you mind if I ask why you went with it over other software? I only went with authentik because I found a tutorial on it first
We expose about a dozen services to the open web. Haven’t bothered with something like Authentik yet, just strong passwords.
We use a solid OPNSense Firewall config with rather fine-grained permissions to allow/forbid traffic to the respective VMs, between the VMs, between VMs and the NAS, and so on.
We also have a wireguard tunnel to home for all the services that don’t need to be available on the internet publicly. That one also allows access to the management interface of the firewall.
In OPNSense, you get quite good logging capabilities, should you suspect someone is trying to gain access, you’ll be able to read it from there.
I am also considering setting up Prometheus and Grafana for all our services, which could point out some anomalies, though that would not be the main usecase.
Lastly, I also have a server at a hoster for some stuff that is not practical to host at home. The hoster provided a very rudimentary firewall, so I’m using that to only open necessary ports, and then Fail2Ban to insta-ban IPs for a week on the first offense. Have also set it up so they get banned on Cloudflare’s side, so before another malicious request ever reaches me.
Have not had any issues, ever.
Have also set it up so they get banned on Cloudflare’s side, so before another malicious request ever reaches me.
How did you end up setting that up?
Fail2ban allows you set different actions for different infringements, as well as multiple ones. So in addition to being put in a “local” jail, the offending IP also gets added to the cloudflare rules (? Is that what its called?) via their API. It’s a premade action called “cloudflare-token-multi”
that’s awesome. thanks!
So among my services I self host, a few need to be publicly accessible for work. For those I wish to remain private, Caddy only allows private IP ranges, plus then Authelia as auth which is set to 30 days. There is then the login of each service behind Authelia as well. It’s as good as it needs to be for my needs.
If I were only self hosting private services, then as others have said, I would put all access through a VPN.
Edit: I should add that of course the private services are then only accessed via VPN to the router (part of the private IP ranges). Caddy as reverse proxy also obfuscates the subdomain names I use.
Caddy only allows private IP ranges
Do you mind telling me more about this? How does that work; a VPN?
Sure, so I use Caddy as a reverse proxy for all my subdomains, the public ones direct straight to whatever service(s) are on IP:port etc, then the private ones only allow private IP ranges of which one is my VPN subnet, therefore only allowing LAN and VPN access. I then also have a section for each of the private subdomains with Authelia authentication which is omitted here in the caddyfile example:
(allowed) { @allowed client_ip 192.168.1.0/24 192.168.10.0/24 192.168.20.0/28 } sub.domain.com { import allowed handle @allowed { reverse_proxy 192.168.80.8:8080 } handle { abort } }
I’ve half ass thought about this but never have tried to actually self host. If you have access to all devices, why not just use your own self signed certificates to encrypt everything and require the certificate for all connections? Then there is never a way to log in or connect right? The only reason for any authentication is to make it possible to use any connection to dial into your server. So is that a bug or a feature. Maybe I’m missing something fundamental in this abstract concept that someone will tell me?
If you have access to all devices, why not just use your own self signed certificates to encrypt everything and require the certificate for all connections?
Sounds like you are describing a VPN. I was using that setup before but small stuff like immich album sharing via a link won’t work properly. Also, having to ensure a vpn is on and connected is a little to much to ask of my partner; they would turn it off and forget about it and then ask why their app wasn’t working :/
I mean more like a self signed TLS certificate with your own host manually set in the browser. Then only make the TLS port available, or something like that. If you have access to both(all) devices, you should be able to fully encrypt by bruit force and without registering the certificate with anyone. That is what I do with AI at home.
I don’t know much about certificates, but doesn’t that just alert the browser that the certificate is not trusted and you can decide if keep going or not?
Sorta, you have to install your certificate authority into the browser and it might complain about verifying that but it will still connect with the encryption.
No no, what I meant is that if I connect to your server without the certificate installed don’t I just get the warning and I can still get through?
Not unless an http port is open too. If the only port is https, you have to have the certificate. Like with my AI stuff it acts like the host is down if I try to connect with http. You have to have the certificate to decrypt anything at all from the host.
If you are just using a self signed server certificate anyone can connect to your services. Many browsers/applications will fail to connect or give a warning but it can be easily bypassed.
Unless you are talking about mutual TLS authentication (aka mTLS or two way ssl). With mutual TLS in addition to the server key+cert you also have a client key+cert for your client. And you setup your web server/reverse proxy to only allow connections from clients that can prove they have that client key.
So in the context of this thread mTLS is a great way to protect your externally exposed services. Mutual TLS should be just as strong of a protection as a VPN, and in fact many VPNs use mutual TLS to authenticate clients (i.e. if you have an OpenVPN file with certs in it instead of a pre-shared key). So they are doing the exact same thing. Why not skip all of the extra VPN steps and setup mTLS directly to your services.
mTLS prevents any web requests from getting through before the client has authenticated, but it can be a little complicated to setup. In reality basic auth at the reverse proxy and a sufficiently strong password is just as good, and is much easier to setup/use.
Here are a couple of relevant links for nginx. Traefik and many other reverse proxies can do the same.
Oh, I really didn’t know that. Thanks
oh, my mistake. tbh, I don’t know enough about it but I’m interested. Why set up a TLS cert for AI at home? How is that benefiting you and your setup?
I’ve seen some people set up SSL certs for self hosted services and not make them publicly available but I didn’t get around to seeing why they were doing it
deleted by creator
Beside the fact that you would like to understand if you’ve done everything properly (that’s good, but I can’t help you here), a VPN on a smartphone can be always active. Mine is always on and I’ve never noticed any battery problem. If you prefer something simpler there’s Tailscale.
If it were only me using the apps, I’d be using a VPN. Over the years, I’ve used OpenVPN, Wireguard, and now Tailscale. In my experience, they work like 99% of the time. That last 1% though is weird connection issues; usually when switching between WiFi and cellular (or vice versa) but sometimes it’s my server or ISP and I have to turn the VPN off and back on to troubleshoot. During those rare times, my partner will either turn off the VPN and forget to turn it back on or they will forget about the VPN completely and not be able to use their phone. Ideally, I’d like to set something up that doesn’t require any potential troubleshooting on their part so I can avoid hearing “why can’t we just use Google photos?” or “what’s wrong with Google home?” 😓
Yes this is the main reason for me. If you’re alone then you don’t care that things occasionally don’t work. Once you have at least one more person or potentially the extended family it’s a whole different story. And then in my opinion a potentially not 100% secured publicly accessible immich instance at home is magnitudes better than having the family just use google photos.
Because like you say, every little hick up from your site is met with “why can’t we just use $bigtech instead, it always works”.
Some of these you’re already doing, but writing a complete* list. *almost garuanteed not to be complete, suggestions welcome
- Have everything behind the same reverse proxy, so that you have only one endpoint to worry about. Run it through ssllabs or similar to check your config.
- On your reverse proxy, add one or more layers of authentication if possible. Many possibilities here: If one app supports client certificates, while another has limited capabilities, you could probably tie together something where IPs are whitelisted to the ither services based on that certificate auth.
- Geoblock all countries you won’t be accessing from
- crowdsec is pretty nice, this detects/blocks threats. kinda like fail2ban but on steroids.
- if you use one of those 5$/month VPSes, with a VPN tunnel to your backend services, that adds one layer of “if it’s compromised, they’re not in your house”.
lastly consider if these things need to be publically avilable at all. I’m happy with 95% of my services only being available through Tailscale (mesh VPN, paid service with good enough free tier, open source+free alternatives available), and I’ve got tailscale on all my devices
-
check
-
check
-
check
-
I saw someone else recommend crowdsec. I’ll look into it, thanks
if you use one of those 5$/month VPSes, with a VPN tunnel to your backend services, that adds one layer of “if it’s compromised, they’re not in your house”.
I’ve heard this mentioned before but I don’t really understand how this works in practice. If the VPS was compromised, couldn’t they use the VPN to then connect to my home?
-
Why? Not every service is meant to be exposed to the open internet. Immich is the only one of what you listed that makes sense to have out in the open.
I’ve been playing around with the voice assistant stuff in homeassistant and it seemingly needs a public url to get all the features. I could be wrong about that though?
I put authentik in front of immich to handle authentication so that I would need need a 2FA code
Most definitely does not need a public URL for Assist in HA. Not sure where you read that.
It sounds like you need a VPN to your internal services if you’re concerned about security.
Most definitely does not need a public URL for Assist in HA. Not sure where you read that.
You’re probably right. At one point, I had a subscription to homeassistant cloud a few years back to use a google nest speaker at the time. I was just going off that I guess. I’ll do some testing and will probably put it back behind tailscale. thanks for the heads up
It sounds like you need a VPN to your internal services if you’re concerned about security.
I’m more so concerned that I set something up incorrectly and would like to be made aware of it in the event someone else noticed
That’s very specific to individual services
What about home assistant? Me and the Family quite often use the HA app on the go, sometimes even from other computers like at my parents or in a hotel to check on the house and the cat. I also gave my dad access to it so he can see if we’re at home and things like that.
Same with my dads HA.
HA had 2 security audits. I would not worry too much. Always depends on what you can control with it. https://www.home-assistant.io/blog/2023/10/19/security-audits-of-home-assistant/
Set up a VPN to get back inside your network if you’re outside it. Then you don’t need HA (or anything else for that matter) to be public.
☝️
