Editing Introduction to a Self Managed Life: a 13 hour & 28 minute presentation by FUTO software

'''Welcome!''' This page is very large and there are many steps, please take your time. If you would like to edit (you're welcome to fix errors even without an account!), [[FUTO:About|please read this first]]. Thank you!
__TOC__
<span id="preface"></span>
= Preface =

<span id="dedication"></span>
== Dedication ==

Thank you to Tim Gilles, aka Slipperman, whose remarks on what makes someone a ''“real professional”'' stuck with me for a lifetime. I listened to Tim on the mixerman radio show. Tim wanted to demystify his craft in a way that anyone could understand; that would inspire EVERYONE to pick up a microphone &amp; a tape machine and give it a shot themselves. He did this with his own [https://archive.org/stream/SlippermansRecordingDistortedGuitarsFromHellreadableVersion/Slipperman%27s%20Recording%20Distorted%20Guitars%20From%20Hell%20%28readable%20version%29_djvu.txt “unique” writing style]. His work inspired me to do the same with everything I’ve done, from board repair to self-managed servers. Tim passed away two years ago. I hope his legacy lives on through everyone who tries to open doors for the next generation rather than gatekeep information via ego inflating elitism.

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:tim_gilles.jpg
</gallery>

<span id="intro"></span>
== Intro ==

I started using GNU/Linux in 2002, back when I saved up the $79.99 necessary to buy SuSE Linux 8.1 Professional as a boxed set from the Best Buy across the street from the Staten Island mall for my 14th birthday. I started hosting my own servers in 2005, and put together systems for my own business’ use since early 2011. I didn’t do everything outlined here immediately; it was slowly built piece by piece over a long time. I never documented it in a way that would allow my grandma to use it. In 22 years, I can’t remember reading GNU/Linux documentation that felt like it was designed for normal people. That’s what I’m looking to do here.

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:suse.jpeg
</gallery>

From 2002 to the present, two things remain true:

* You can do cool things with GNU/Linux.
* These cool things are hidden behind a labyrinth of
** Half baked software.
** [https://medium.com/@fulalas/gnome-mess-is-not-an-accident-4e301032670c Horrible UI].
** Forum elitists &amp; [https://forum.netgate.com/topic/184398/kea-dhcp-missing-register-dhcp-leases-in-dns-resolver/7 gaslighting assholes] who will make you think YOU’RE the crazy one for expecting [https://forum.netgate.com/post/1174947 things to work].
** People that will tell you to ''“RTFM”'' with no regard for whether that documentation actually works.
** [https://wiki.futo.org/index.php/File:Lu55028jxckj_tmp_1514c051.png black boxes]. I mean '''''literally hidden''''' behind ''actual black boxes.'' For six months. [https://bugs.launchpad.net/subiquity/+bug/2062102 Unfixed]. On the stable version of a server operating system'' (that bug is present in 24.10 long-term-stable even today)''.

So much of the open source user experience is '''not designed for normal people'''. Whether it was using [https://bbs.archlinux.org/viewtopic.php?id=21650 NDISwrapper 20 years ago to get wifi to work] or messing with [https://bbs.archlinux.org/viewtopic.php?id=1718 SCSI emulation] to burn a CD, GNU/Linux is pain. It’s all pain.

It’s painful enough that people will happily trade their data, sovereignty, privacy, and their rights to avoid ever having to deal with it; and I can’t blame them.

This has to change. As of 2024, most of you live your life:

# Dependent on closed source software.
# Running on someone else’s server where you can be kicked off at any time.
# Forced into [https://techcrunch.com/2024/03/05/roku-disables-tvs-and-streaming-devices-until-users-consent-to-forced-arbitration/ forced arbitration] or your device won’t work anymore.
# With [https://www.nytimes.com/2022/08/21/technology/google-surveillance-toddler-photo.html no privacy].
# [https://web.archive.org/web/20240628071953/https://www.nytimes.com/2024/06/26/technology/terms-service-ai-training.html Training AI] with your creations.

Now is a time like no other for you to feel empowered to build systems that '''you''' control &amp; understand.

My goal with this guide is not to tell you the way you '''HAVE''' to do something, or to imply that my way is the best. My goal is to inspire you by showing you what’s possible. You don’t have to be a computer engineer or someone with an IQ of 160 to figure this all out. And, admittedly, to inspire capable developers to look at the pain points scattered throughout this guide (of which there are many) and decide ''“enough is enough; let’s make this better”''.

The fun here is in ''building your own system, your own way''. This is my sovereign cloud; there are many like it, but this one is mine. I can’t wait to see how you build yours.

<span id="why-build-your-own-sovereign-cloud"></span>

= Why Build Your Own Sovereign Cloud? =

Apple and Google push users into closed ecosystems while removing options for personal control over data. Think back to when smartphones had microSD card slots, so you could store your photos, videos, &amp; music locally &amp; cheaply. As these companies started pushing paid cloud services, microSD slots disappeared from every phone. Apple no longer [https://www.macrumors.com/2024/05/15/ios-17-5-bug-deleted-photos-reappear/ gives you a working “delete” button], and Google has [https://www.nytimes.com/2022/08/21/technology/google-surveillance-toddler-photo.html mistakenly flagged people as criminals for sending photos a doctor requested of their sick child] during COVID lockdowns. These issues come up because you don’t own the software or services you’re using. If you can’t review the source code, it’s not your software. If you can’t host the service yourself, it’s not really yours.

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:lu55028jxaty_tmp_bb449a5f.png
File:lu55028jxaty_tmp_1004238c.png
</gallery>

FUTO is looking to change that. We want to provide solutions that let you take back control, whether it’s running your own cloud or hosting your own services. Many of these services have 1% adoption (if they’re lucky!) because of the barriers to use.

One example is Immich; it’s photo gallery software that uses local AI, so you never have to worry about your personal data being analyzed by some remote server. It’s incredibly fast &amp; efficient! I think it’s the best in its field. Right now, if you want to use it, you need to set up your own GNU/Linux server and use Docker to get everything running. You either become a GNU/Linux sysadmin or you sell your data (and your soul) in exchange for a half-decent UI.

'''Until now!'''

<span id="futos-belief-in-self-managing-your-own-servers."></span>
== FUTO’s belief in self managing your own servers. ==

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:cluelessimmich.png
</gallery>

We believe that any piece of software we create or offer that has a client, must be accompanied by server source code that allows you to run your own server. You have to have control over your devices. At the same time, if we throw the source code at you and tell you ''“have fun!”'', have we really enabled you to run your own system? That’s akin to throwing a party and saying ''“hey, anyone who wants to join us is allowed in!”'' when you only tell your best friends where the door is. We want the door to the party to be open to everyone; and for '''''all of you''''' to know where it is. So, let’s see if we can put spicy brownie’s concerns to rest.

<span id="the-rabbit-hole-to-hell"></span>
=== The Rabbit Hole to Hell ===

I’m going to show you exactly how to set this up because that’s been a common question in the comments. I’m going to show you how to set up <code>Immich</code>. To do that, I need to show you how I get my files from my phone to my server. If I’m doing that, I’m connecting to my server from outside, which means I have to show you how to set up a VPN tunnel. I’m not going to forward ports for all these random services. If I’m doing that, I might as well show you how to set up a router that will always get updates, which means building your own.

While I’m at it, I might as well show you how to block all ads, even when you’re connected from your phone. While we’re in there, let’s show you how to set up something similar to Google Docs, Google Sheets, calendar, contacts, home surveillance with notifications, self-hosted mail, a business phone system that curses out annoying customers ''for'' you, and everything else.

<blockquote>'''Warning:''' This becomes a rabbit hole very quickly because there are so many items to cover. I’m not going to breadcrumb you. I want to provide you with everything, which means we have to start from the BEGINNING!
</blockquote>
<span id="a-long-journey-ahead"></span>
== A Long Journey Ahead ==

This isn’t going to be a 10-minute video, nor will it be a 10-page guide. It’ll probably be a ten-hour video, and a 1000-page guide. You’ll get to figure out how much I hate you based on whether or not I provide you with timestamps or a table of contents.

<span id="understanding-the-basics-modem-router-switch-and-wireless-access-point"></span>
== Understanding the Basics: Modem, Router, Switch, and Wireless Access Point ==

Before we dive into discussing building a router, I want you to understand the key components of your home network: the modem, router, switch, and wireless access point. These devices work together to connect you to the internet and allow multiple devices to communicate with each other. Most consumer products package the router, switch, and wireless access point all in one, hiding from you what each component is for. You might even have a modem that includes all three, meaning you have one device on your home network! Let’s break down the purpose of each device.

<span id="modem"></span>
=== Modem ===

The modem is your gateway to the Internet, connecting your home to your Internet service provider (ISP).

'''What a Modem Does:'''

* Converts the long-range signal from your ISP (e.g., cable, fiber, DSL) into a short-range  signal that your devices can use (eg Ethernet, Wi-Fi).  Short range signals are helpful because they are safer and because they can be used with simpler, cheaper electronic components.
* Reformats the encoded data from the signal to a format understood by your devices.
* Acts as the bridge between your ISP’s network and your home network.

'''Types of Modems:'''

* '''Cable Modem:''' Connects to your ISP via a coaxial cable.  It sends (and receives) electromagnetic signals through that cable.  (Like the signals used by news-radio stations, but different frequencies, and forced into a cable, rather than broadcast on the air).
* '''DSL Modem:''' Connects via a "twisted pair" cable -- the kind used for landline telephones.  Uses electromagnetic signals, like a cable modem, but different frequencies.  Be careful of telephone cables: they typically also carry electrical power, to power a landline telephone.  The voltage can be high enough to be painful.
* '''Fiber Modem:''' Connects via a fiber-optic cable.  More properly called an optical network terminal (ONT).  It transmits light (photons).  Inside, the cable is transparent, like plastic.  Don't point the end of the cable at your eyes: depending on the type of light used, the light can be invisible yet have enough energy to damage your eyes and blind you.
* '''Cellular Modem:''' Connects wirelessly, via the nearest cell tower -- there is no cable going to the ISP.  This is the exact same technology used by a cellphone but the ISP provides a box that looks like a DSL modem and that you place, typically, near a window, to optimize reception.
* '''Dialup modem:''' Connects via a "twisted pair" cable.  It uses only the tiny range of electromagnetic signals that telephone technology uses to represent sound.  Dialup modems often have a tiny speaker, which lets you hear the signal, for troubleshooting.  Dialup was the first technology offered by ISPs.  It is now extremely rare, because modern technologies are much much faster.

<blockquote>'''Important:''' A modem typically has only one Ethernet port, which is why you need additional devices like routers and switches to connect multiple devices in your home. A modem may have a phone jack to attach a standard telephone.
</blockquote>
<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:lu55028jxaty_tmp_6d85bda7.png
File:lu55028jxaty_tmp_60f34d25.png
File:lu55028jxaty_tmp_3f4c8a0.png
File:lu55028jxaty_tmp_40634abb.png
</gallery>

<span id="router"></span>

=== Router ===

The router manages traffic between your local network (your home devices) and the internet (outside world).

'''What a Router Does:'''

* Allows you to have more than one device on your network.
* If you attach your computer to your modem directly, you are simply connecting to the “outside” world’s network. This is referred to as '''“WAN”''' - Wide Area Network is a network that connects multiple LANs over large distances, while a LAN is a network confined to a local area. This can work, but when you do this you do not have an internal network. The computer you attached to your modem is the only computer in your home that can go online with this configuration.
* Routers create a 2nd internal network for your devices so you can attach more than one thing to the internet (WAN). Wouldn’t it suck if you could only have one wired device attached to your home internet? This is why most people need a router!
* '''Routes Traffic:''' Directs internet traffic from the OUTSIDE (this is called the “WAN”) to the correct device on the INSIDE, your home network (this is called the “LAN”), and vice versa. Now, multiple devices (e.g., computers, phones, smart TVs) can communicate with the internet through your modem, and with each other within your home.
* '''Provides NAT (Network Address Translation):''' Translates your devices’ private IP addresses into a single public IP address provided by your ISP.

<blockquote>'''Note:''' The router you get from your ISP or buy from a store, 99% of the time, is a combo device that looks like this: includes a router, switch, and wireless access point all in one box. Understanding their roles separately is key when setting up a more advanced system like '''pfSense'''.
</blockquote>
<span id="traditional-wired-router"></span>
==== Traditional wired router: ====

Below is a traditional wired router. This combines a router &amp; a switch but has no wireless access point.

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:lu55028jxaty_tmp_62add313.png
</gallery>

<span id="cheap-walmart-wifi-router"></span>
==== Cheap Walmart Wi-Fi router: ====

This is a TP-Link wireless router: a router, switch, and wireless access point all in one. This is most likely what you have in your closet right now, covered in wires, under the set of workout pants you bought six months ago after your failed New Year’s resolution to go running every morning. It has slow speed unless you’re 2 feet from it. These often come with SIP-ALG (a component that transforms Voice-over-IP packets, which generally isn't needed any longer today) on by default, and will mess with your phone systems endlessly even if you try turning it off. Avoid the Walmart routers.

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:lu55028jxaty_tmp_2a9c98fc.png
</gallery>

<span id="switch"></span>

=== Switch ===

A switch expands the number of devices you can connect to your local network using Ethernet cables.

'''What a Switch Does:'''

* '''Expands Connectivity:''' If your router only has a few Ethernet ports, a switch allows you to connect more wired devices (e.g., computers, gaming consoles, network-attached storage).
* '''Forwards Data:''' A switch is smarter than a basic splitter. It knows which devices are connected to each port and forwards data to the correct device, improving network efficiency.
* The type of basic switch I am using for this example is the smaller type below, that has no advanced routing features, settings, or web interface to mess with. It’s just a dumb switch.

Switches come in different sizes, from small 4-port models to large 24-port (or even larger) models used in business environments. The small Netgear switches that cost $15 are more than adequate for most people’s home networks &amp; will not cause random disconnects or issues with our router setup.

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:cheap_netgear_switch.jpg
</gallery>

<span id="cheap-switch"></span>
==== Cheap switch ====

This is a [https://www.amazon.com/NETGEAR-5-Port-Gigabit-Ethernet-Unmanaged/dp/B07S98YLHM basic Netgear switch that you get for $15]. It allows you to connect four devices to your pfSense router. You would attach the LAN port on the pfsense router to a port on this switch ''(any port is fine)'' &amp; then connect your wired devices ''(wireless access point for wifi, computers, etc)'' to other ports on the switch. Some points to note:

<ul>
<li><p>This switch is gigabit - meaning, 1 gbps.</p>
<ul>
<li>1 gbps = stuck transferring around 100 megabytes per second real world performance (aka the speed of ten year old hard drives).</li>
<li>This means even if you have a fast solid state drive in the server &amp; your personal computer, transfer speed will be around 100-120 megabytes per second.</li>
<li>If you have a gigabit internet connection &amp; are downloading a file at 1 gbps, you can also grab a file from your server without slowing your download.</li></ul>
</li>
<li><p>This has no Power over Ethernet (PoE)</p>
<ul>
<li>If you want to power wireless access points, office voice over IP (VoIP) phones, or cameras, you have to plug them into something or get a PoE injector later.</li>
<li>A Power over Ethernet switch can power devices you plug the ethernet cord into which is very cool for setting up security cameras, because you only have to run 1 wire to each camera.</li>
</ul>

<p>These cheapies will usually not have Power over Ethernet to power cameras &amp; wireless access points &amp; office desk phones, nor will they usually support configuring ports for VLANs ''(we will get into that in the wifi section at the end)''. This is a good ''starter'' switch since it is reported to pass VLAN tags, so if you bought wifi access points or switches that supported creating isolated networks this switch would pass those tags ''(we’ll get into that at the end of the guide); no need to worry about that right now.''</p>
<p>These cheap switches work great, and also come in 8 port versions for a few bucks more.</p></li></ul>

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:expensive_netgear_switch.jpg
</gallery>

<span id="expensive-switch"></span>

==== Expensive switch ====

The [https://www.bhphotovideo.com/c/product/1383572-REG/netgear_xs724em_100nas_24_port_10_gigabit_multi_gigabit_ethernet_smart.html?ap=y&smp=Y Netgear XS724EM switch] is an expensive, fancier switch.

* Speed
** Supports 2.5 gigabit or 10 gigabit per second Ethernet on its ports.
** If you have a network interface card (NIC) that supports 2.5 gbE on each end ''(these are becoming more common)'', you can get over 270 megabytes per second transfer rate ''(more than 2x a normal gigabit switch)''
** If you have a network interface card (NIC) that supports 10 gbE on each end ''(your computer does not have this unless you bought it separately &amp; installed it)'', you can get over 800 megabytes per second even with a poorly tuned setup. This is likely faster than any of the drives inside your computer unless you bought fancy NVMe drives.
* Power
** Can supply power to a bunch of cameras, phones, wireless access points over Ethernet cables (PoE). Has the advantage that cameras in hard to reach spots only need an Ethernet Cable and not an additional Power Cable since the neccesary Power is transmitted to the camera via the Ethernet Cable.
* Ports
** Has 24 ports instead of 5, can connect a lot more stuff.
** '''COMPATIBILITY''' of the ports - does 10 GbE over standard ethernet plugs/jacks. If you wire your house with [https://www.monoprice.com/product?p_id=13072 good cat6a] and put [https://www.ebay.com/itm/235729074315 good 10 gbE network interface cards] in the machines you’re working with you can get 800 megabyte per second networking all around your house without digiorno connectors.
* Virtual LAN support
** Allows you to create separate networks on the same switch.
** Can keep your untrusted internet of things (IoT) devices like cheap light bulbs &amp; cameras &amp; thermostats on isolated separate networks from your trusted devices.

The VLAN support is a big one because later on when we connect wifi access points that are advanced we are going to do far more than just make a “guest network”; we are going to make a network that your IoT devices (bulbs, thermostats, cameras, etc) can connect to and isolate them in a way where your computer running your security cameras &amp; home automation can connect to them, but they are isolated from connecting to anything else. This isn’t necessary though and a bunch of you will probably skip the VLAN part at the end, since that gets a little too complicated for a home setup.

This is an expensive switch. There are many in between, but I thought it’d be useful to show an example of the cheap side &amp; expensive side to show what is available &amp; what you can get for the money. If you are ok with gigabit ethernet you can easily get by with way cheaper; right now you can either buy gigabit switches cheaply, or 2.5 gbe &amp; 10 gbe switches at crazy high prices. There isn’t much in between.

<span id="wireless-access-point-wap"></span>

=== 4. Wireless Access Point (WAP) ===

A wireless access point (WAP) provides Wi-Fi access to your network, allowing devices like phones, tablets, and laptops to connect wirelessly. You could add a wireless access point like the ones below, to the old blue Linksys router above, to turn it into a “wireless router”.

'''What a Wireless Access Point Does:'''

* '''Provides Wireless Connectivity:''' The WAP connects to your router (or switch) and broadcasts a Wi-Fi signal, letting wireless devices connect to your network.
* '''Doesn’t Route Traffic:''' It’s important to note that a WAP doesn’t perform the same function as a router. It simply extends your network by adding wireless connectivity.

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:lu55028jxaty_tmp_e4850c8b.png
</gallery>

These are mesh network access points. They allow you to connect each to your switch and place them in separate areas of your home to make sure you have great connectivity everywhere.

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:tplink_mesh_example.jpg
</gallery>

The way these work is you would place the access points on different parts of your house and have an ethernet wire going to each one. The access points intelligently work together to figure out which one you should be connected to based on which provides the strongest signal to your laptop/phone where you are right now. You’d place one on the side of your house, one in the basement, one on each side of each floor in your home, and wire them all to your switch &amp; you’ll get amazing wifi connectivity from anywhere. Good wireless access points will switch over so seamlessly that your file transfer does not stop or fail as it is happening.

These setups are more expensive since proper mesh equipment that works right costs more &amp; you are buying multiple access points.

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:lu55028jxaty_tmp_62add313.png
</gallery>

This is an ancient wired router with no wifi.

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:lu55028jxaty_tmp_dad6faba.png
</gallery>

This is a cheap ass wireless access point. I don’t recommend any of these especially when something like a [https://www.ebay.com/itm/116401635619 TPLink EAP6120] is about $50 used &amp; offers much better seamless roaming if you want to add access points later, VLAN functionality, etc. I know it’s tempting to buy the lame ones because they are in stock at best buy &amp; walmart for instant gratification but you’ll regret it later.

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:lu55028jxaty_tmp_4bc4c5a.png
</gallery>

This is an ancient wireless router that is a legend. The unbreakable, unbeatable, Linksys WRT54G. It is a router, a switch, and a wireless access point all in one.

<span id="internet-protocol-addresses"></span>

=== Internet Protocol addresses ===

You have an address on the front of your building. You have a phone number - this is how people find you. Your router will be how you get an IP address from your internet service provider. It usually looks like <code>64.91.255.98</code> or <code>8.8.8.8</code> - you may have seen this before.

Most of you with a home internet connection have something called a '''Dynamic IP'''. This means that your IP can change.

Your IP address may change for a number of different reasons:
* When you unplug your modem [for a long period of time].
* When you plug your modem into a new router.
* Every day, just for the hell of it!

This can make things more difficult than when you have a '''static IP''' - static IPs do not change. You get an internet protocol address, and that’s what you’re stuck with, for better or for worse.

For home users, most people don’t need a static IP. Static IPs are for when I want something to ''“stay put”''. I want my phone number to stay put so people know where to find me. I want my home address to stay put so the mailman knows where to find me ''(and so I know where to go home!)'' and, in this case, I want my IP to stay put so I can always find my home server, no matter where I am in the world.

If you are reading this - you likely have a '''dynamic IP''' provided by your home internet service provider. We will have a workaround for this that allows you to be able to find your server at the same place every time you go to use it no matter how often its IP changes.

<span id="how-these-devices-will-work-together-in-your-setup"></span>

== How These Devices Will Work Together in Your Setup ==

For this setup, you’ll use a dedicated '''pfSense''' router instead of the combo device provided by your ISP. Here’s how the connections work:

<span id="modem-to-router"></span>
=== 1. Modem to Router ===

* The modem takes the signal from your ISP and passes it to your '''pfSense''' router via an Ethernet cable.
* The modem will be connected to the WAN (Wide Area Network) port on the router.

<span id="router-to-switch"></span>
=== 2. Router to Switch ===

* Your '''pfSense''' router manages traffic between your devices and the internet.
* Since the Intel NUC running '''pfSense''' has only two Ethernet ports, you’ll connect the second port (the LAN (Local Area Network) port) to a switch to connect multiple devices.

<span id="switch-to-devices"></span>
=== 3. Switch to Devices ===

* The switch is connected to the LAN port of your '''pfSense''' router.
* Any wired devices (like computers, gaming consoles, or network storage) can be connected to the switch using Ethernet cables.
* This allows multiple devices to communicate with each other and access the internet through the '''pfSense''' router.

<span id="adding-wireless-access"></span>
=== 4. Adding Wireless Access ===

This will allow your phones, laptops, and other wireless devices to connect to the network without wires.

* If you only plan to have wireless devices on your network, you can attach your wireless access point directly to the LAN port on your '''pfSense''' router.
* If you wish to have a combination of wired &amp; wireless devices on your network, you would attach a wired switch to the LAN port on your '''pfSense''' router, and then plug the Wi-Fi access point into a port on your switch.
* If you have no plans to have wireless devices on your network, you do not need a wireless access point.

<span id="a-common-home-network-setup-vs.-your-new-setup"></span>
=== A Common Home Network Setup vs. Your New Setup ===

'''Common Setup (with ISP Combo Device):'''

* Modem → ISP-provided combo device (modem + router + switch + WAP)
* All devices (wired and wireless) connect to the combo device.

'''Your New Setup (with pfSense):'''

* Modem → '''pfSense''' Router (dedicated firewall/router)
* '''pfSense''' Router → Switch (for wired devices)

<blockquote>This new setup gives you better control over your network, improved security, and the ability to block ads with pfSense and tools like pfBlockerNG. It is important that you know what each component does &amp; their purposes. By understanding what each component does, you’ll be better equipped to set up and manage your new pfSense-based network!
</blockquote>
<span id="why-build-your-own-router"></span>

= Why Build Your Own Router? =

<span id="regular-security-updates-openvpn"></span>
=== Regular Security Updates &amp; OpenVPN ===

Let’s start at the very beginning with OpenVPN. We are not opening ports to the internet for ANYTHING, except for receiving self-hosted mail. We’re running a bunch of different open source services that less than 0.1% of the population (if I’m being generous) actually use. I '''LIKE''' <code>Immich</code>, <code>Home Assistant</code>, <code>Syncthing</code>, <code>FreePBX</code>, <code>OnlyOffice</code>, <code>Nextcloud</code>, <code>Mailcow</code>, <code>Frigate</code>. But I don’t want them just open to the internet.

They’re nice software, but they’re [https://wiki.futo.org/index.php/FUTO:General_disclaimer used by 0.0001%] of the population. Further, even if they WERE secure, by opening ports to the internet, I am letting every Tom, Dick &amp; Harry who wants to peek in see what I am running on my IP address.

OpenVPN is used by companies in the S&amp;P 500, banks, and governments; it’s everywhere! The beauty of OpenVPN is that if there’s ever a security breach, it’s going to get found and fixed because there are tens of millions of eyes on it at any given moment. There is too much ''investment'' in OpenVPN for it to wither on the vine and become fundamentally insecure. OpenVPN is as secure as it gets, and while it’s not ''perfect'', we are massively reducing our ''RISK'' of being hacked &amp; exploited by utilizing OpenVPN to get into our home network vs. opening ports willy-nilly to 10 different pieces of software.

I don’t want people to be able to see that these services are all running on my server. That means there are four, six, eight, or 15 different points of failure. I’d rather have one point of failure that’s managed properly. And that’s what a VPN is for—a way to create a secure, encrypted tunnel between your phone and your server.

<span id="why-cant-i-buy-a-30-router-at-walmart"></span>

== Why can’t I buy a $30 router at walmart? ==

<span id="short-lifespan-for-firmware-updates"></span>
=== Short lifespan for firmware updates ===

Consumer routers you find in stores may offer features like OpenVPN, but the problem is that many [https://www.bleepingcomputer.com/news/security/netgear-leaves-vulnerabilities-unpatched-in-nighthawk-router/ stop receiving updates shortly after you buy them].

<span id="buggy"></span>
=== Buggy ===

Many of the lower end store routers are buggy and can cause problems with what I am showing you how to set up. Certain TP-Link routers have randomly messed with SIP traffic in the middle of a call, and the router that Spectrum and Verizon provide have <code>SIP-AlG</code> turned on by default; which will mess with our phone system. They don’t let you turn it off in the configuration settings either!

Back to my point; using a router where you are at the mercy of the manufacturer to provide you with updated firmware leaves you vulnerable to security risks as new exploits are discovered. For example, three years down the line, there might be a very important update for OpenVPN, but your router’s manufacturer might have stopped supporting your model after just six months. Now you’re screwed.

<span id="increased-likelihood-of-getting-hacked-over-time"></span>
=== Increased likelihood of getting hacked over time ===

You’re making it harder for yourself by using a router that will become vulnerable to exploits in OpenVPN. OpenVPN is exceptional software: these holes get plugged, and they get plugged fast.

…if the manufacturer actually updates the firmware. They often don’t. Think about it:

# You already paid for the router.
# Providing you with updated firmware costs them money &amp; time.
# But they already have your money.
# So they don't care.

'''You might think I’m being bombastic; what’s so bad about using an older version of OpenVPN?'''

<span id="openvpn-exploits"></span>

=== OpenVPN exploits: ===

A '''CVE''' is a common vulnerability &amp; exploit - aka, a way to hack into something. These are a small number that have occurred over the years. Finding CVEs isn’t a bad thing, every piece of software ever created is going to have security vulnerabilities. It is only bad if you are running hardware that you cannot update once a fix has been released.

<span id="cve-2024-27459-cve-2024-24974-cve-2024-27903-cve-2024-1305"></span>
==== 1. CVE-2024-27459, CVE-2024-24974, CVE-2024-27903, CVE-2024-1305 ====

* '''Discovered''': March 2024
* '''Description''': Multiple vulnerabilities were found, mainly affecting OpenVPN’s client-side on Windows, Android, iOS, macOS, and BSD. These included stack overflow, unauthorized access, &amp; plugin flaws leading to potential remote code execution (RCE) and local privilege escalation (LPE). Users were advised to update to OpenVPN versions 2.6.10 or 2.5.10 to mitigate the risks. ''You can only update OpenVPN versions if your router lets you.''

<blockquote>'''Terminology note:''' “client-side” means the part of the software that runs on your device (like a computer or smartphone), as opposed to “server-side”, which would be the part running on a remote server (Apple/Google’s server).

“Remote Code Execution (RCE)” is a vulnerability that lets a hacker run code they want to run on your device. “Local Privilege Escalation (LPE)” means a vulnerability that lets a hacker get higher permissions (i.e. becoming an admin rather than being a regular user) allowing them to do things they shouldn’t or gain full control over your system.
</blockquote>
* '''Sources''':
** [https://cybersecuritynews.com/openvpn-vulnerabilities-rce-attack/ Cybersecurity News]
** [https://openvpn.net/security-advisories/ OpenVPN Security Advisory]
** [https://campustechnology.com/Articles/2024/08/16/Report-Increasing-Number-of-Vulnerabilities-in-OpenVPN.aspx Campus Technology]

<span id="code-signing-key-intrusion-openvpn-2.5.8"></span>
==== 2. '''Code Signing Key Intrusion (OpenVPN 2.5.8)''' ====

* '''Discovered''': December 2022
* '''Description''': An intrusion was detected involving OpenVPN version 2.5.8. There’s no evidence suggesting the key was misused &amp; OpenVPN proactively re-released the software signed with a new key for security. This is why updates matter.
* '''Sources''': [https://openvpn.net/security-advisories/ OpenVPN Security Advisory]

<span id="cve-2022-0547"></span>
==== 3. '''CVE-2022-0547''' ====

* '''Discovered''': February 2022
* '''Description''': Enabled authentication bypass in external authentication plug-ins when more than one of them makes use of deferred authentication replies, '''''which allows an external user to be granted access with only partially correct credentials.''''' aka, I can have a sawed off copy of your house key &amp; still get in.
* '''Sources''': [https://community.openvpn.net/openvpn/wiki/CVE-2022-0547 OpenVPN Community]

<span id="cve-2020-15077-cve-2020-36382"></span>
==== 4. '''CVE-2020-15077, CVE-2020-36382''' ====

* '''Discovered''': 2020
* '''Description''': These vulnerabilities affected OpenVPN Access Server, with risks of information leakage and potential denial-of-service (DoS). Patches were released fast to address these security issues, which requires you have a router that allows you to continue updating it after the manufacturer has given you the middle finger &amp; told you to buy a new one.
* '''Sources''': [https://openvpn.net/security-advisory/access-server-security-update-cve-2020-15077-cve-2020-36382/ OpenVPN Security Advisory]

<span id="cve-2018-9334"></span>
==== 5. '''CVE-2018-9334''' ====

* '''Discovered''': 2018
* '''Description''': A denial-of-service vulnerability in OpenVPN’s handling of authentication processes, potentially allowing attackers to disrupt services was patched.
* '''Sources''': [https://openvpn.net/security-advisories/ OpenVPN CVE List]

<span id="cve-2017-7521"></span>

==== 6. CVE-2017-7521 ====

* '''Discovered''': 2017
* '''Description''': A memory exhaustion flaw was found where an attacker could exploit OpenVPN’s message handling to cause service disruption.
* '''Sources''': [https://openvpn.net/security-advisories/ OpenVPN CVE List]

<span id="guaranteed-long-term-compatibility-updates"></span>
== Guaranteed long term compatibility &amp; updates ==

Even a cheap 10-year-old desktop PC can be a good router for the next ten years; as long as it has a good network interface card. If it runs out of RAM or new network technologies come out, you won’t throw it away; you’ll buy a new network card for $40 or more RAM at a yard sale. Ten years from now, going from 2 GB of RAM to 8 will probably cost less than $10.

Using a standard x86 PC as a router, with known good Network Interface Cards, means you are less likely to encounter compatibility or longevity issues when using any of these open source router systems. It gives you more control, and if you’re reading this, you probably have an old desktop PC in the garage or closet you’re not using anyway. Get it two good network interface cards and get it back in commission!

<span id="what-about-openwrt"></span>
== What about OpenWRT? ==

There are open source packages like [https://openwrt.org/ OpenWRT] doing the lord’s work to keep these routers going. This is a good project, run by good people. I do not want to denigrate them in any way; what I am about to say is in no way their fault. They do their best to keep routers running with their firmware for as long as possible, but eventually, it becomes too difficult or untenable to provide updates for older chipsets &amp; hardware, and they [https://openwrt.org/toh/start fall off the list]. Those old routers will only work with older versions of OpenWRT (especially for those [https://openwrt.org/supported_devices/openwrt_on_432_devices 4/32 devices]).

But it’s a lot of work to support 100s of different makes &amp; models, all using their own specific hardware. When we build a router using a standard computer, we can install router software like '''pfSense''' or <code>OPNsense</code>, which means the chances of our hardware not getting updates/not being supported shrinks to almost nothing. These open source projects do not have to support a gazillion different hardware configurations. They support x86, and if you have x86 (most normal desktop computers are x86), you’re good. It makes it easier to maintain on a mass level &amp; provide ''regular'' updates to. The likelihood of your “hardware not being supported” with an open source router distribution when it is a desktop PC with a good network card shrinks to near 0.

By building your own router using '''pfSense''', an open-source firewall, and cheap, dedicated hardware, you guarantee long-term support and control over your setup. With '''pfSense''', you can get regular updates, customize your network settings, and even block ads across all devices using '''pfBlockerNG'''.

<span id="building-our-own-router"></span>

= Building Our Own Router =

Let’s dive into the first step: setting up '''pfSense''' on an '''Intel NUC''' (a small-factor barebone PC, [https://en.wikipedia.org/wiki/Next_Unit_of_Computing Next Unit of Computing]) to serve as your router. We’ll be setting this up with OpenVPN, which is very important for connecting securely to your home network.

As for the hardware, I’m using an Intel NUC because it’s compact, reliable, and it has two Ethernet ports, which are necessary for setting up a router. One port is used for your '''WAN''' (internet), and the other for your '''LAN''' (internal network). For a '''pfSense''' router, we must choose a machine with '''TWO''' ethernet ports, not one!

<span id="why-pfsense"></span>
== Why pfSense? ==

I chose '''pfSense''' ten years ago because:

# It’s open-source.
# It’s fast.
# It gets regular updates for security issues.
# The parent company has paid corporate &amp; business clients relying on their software, which is based on an open source core. The developments with regards to making certain network cards work well with FreeBSD get included upstream to the free versions.
# This means that me, as a scrub who didn’t pay for it, get something that is very similar to what corporate clients who are paying $10,000 or more are getting.
# If I mess something up with my very unusual custom setup, I can pay the developers of the software to fix it for me. This level of support is not common in many open source projects. If I want to cry uncle &amp; pay them an annual fee, they will respond to my questions &amp; provide me with '''''REAL''''' answers rather than tell me to go ''“rtfm”''.
# It comes with features like '''pfBlockerNG''' to block ads, scams, and malware at IP &amp; DNS level with regular updates.

I use '''pfSense''' now because:

# I’m used to it.
# The idea of redoing my complicated setup from scratch gives me hives.
# See #2, in regard to becoming acquainted with the unique quirks of other open source software.

I had very good reasons for choosing '''pfSense''' ten years ago – and I have good reasons to use it today. That doesn’t mean it’s the best. Feel free to use whatever you want to use. For the purposes of this guide, I will be using '''pfSense'''.

There’s a bit of a debate between '''pfSense''' and <code>OPNsense</code>. TL;DR, the developers of '''pfSense''' are [https://opnsense.org/opnsense-com/ not the nicest people sometimes]. If this bothers you, consider checking out <code>OPNsense</code>. Since I’ve been using '''pfSense''' for a decade, I’ve built much of my infrastructure around it. I am well aware of its quirks and don’t feel like setting up my network from scratch, so I am using '''pfSense''' for this tutorial. Regardless of the [https://news.ycombinator.com/item?id=17431809 developers], you are infinitely better off using '''pfSense''' on your own hardware than standard routers.

<span id="choosing-the-right-hardware"></span>
== Choosing the Right Hardware ==

<span id="why-an-intel-nuc"></span>
=== Why an Intel NUC? ===

When searching for hardware to build a '''pfSense''' router, you’ll often come across a variety of mini PCs on platforms like Amazon. However, there are several issues with these options:

# '''Inconsistent Quality:''' You’ll find reputable brands like Mikrotik listed alongside unknown generic random stuff. I trust Mikrotik - I don’t trust random junk. Amazon allows random junk from unverified, untrusted vendors to show up routinely at the top of the search results.
# '''Unreliable Reviews:''' Amazon’s review system has known issues:
#* Reviews from [https://www.youtube.com/watch?v=qZCMislL6_I&list=PLkVbIsAWN2ls4fzQbP9fdW66tjcIs4JNQ&index=5&pp=gAQBiAQB unrelated products (e.g., digital picture frames) applied to air conditioners].
#* Vendors [https://www.youtube.com/watch?v=eS698R-bxuc&list=PLkVbIsAWN2ls4fzQbP9fdW66tjcIs4JNQ&index=4&pp=gAQBiAQB bribing customers for positive reviews] without consequences.
#* Potentially fake or misleading reviews.
# '''Safety Concerns:''' Amazon has a history of selling mislabeled or dangerous products, including:
#* [https://www.youtube.com/watch?v=B90_SNNbcoU&list=PLkVbIsAWN2ls4fzQbP9fdW66tjcIs4JNQ&index=2&pp=gAQBiAQB Incorrectly rated electrical fuses].
#* [https://www.youtube.com/watch?v=y83BS_mK9GE&list=PLkVbIsAWN2ls4fzQbP9fdW66tjcIs4JNQ&index=1&pp=gAQBiAQB Faulty electrical crimps].
#* Litter boxes that [https://www.dailydot.com/news/cat-stuck-in-automatic-litter-box/ kill cats].

…and the list goes on. This guide is going to be 600+ pages when done; do you want to do all of this work only to have the primary component be a piece of junk from a website that sells cat guillotines? No.

<span id="the-better-alternative-repurpose-an-old-desktop-pc"></span>
=== The Better Alternative: Repurpose an Old Desktop PC ===

Instead of risking your project with unknown mini PCs, consider using an old desktop computer:

# '''Reliability:''' A 10-12 year old desktop is likely more reliable than no-name mini PCs.
# '''Choice of Network Card:''' Desktop PCs offer PCI Express slots for additional network cards, so YOU can choose the network interface card for your setup. You often do not know what chipsets are used in the no-name-mini-PCs. '''pfSense''' &amp; other FreeBSD-based routers are sensitive to poor-quality chipsets.
# '''Cost-Effective:''' You can re-purpose an old desktop you already have &amp; save money on purchasing new hardware.

<span id="choosing-the-right-network-interface-cards-nics"></span>

=== Choosing the Right Network Interface Cards (NICs) ===

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:lu55028jxaty_tmp_12bd653b.png
File:lu55028jxaty_tmp_a186c0a7.png
File:lu55028jxaty_tmp_a7b91798.png
</gallery>

To transform your old desktop into a capable router:

# '''Add Quality NICs:''' Install high-quality network cards, preferably Intel-based.
# '''pfSense Compatibility:''' Check the '''pfSense''' forums for compatible chipsets and cards.
# '''Examples of Good NICs:'''
#* Intel X540.
#* Intel 350.

<span id="caution-when-purchasing-nics"></span>
=== Caution When Purchasing NICs ===

# '''Avoid Realtek at all costs:''' Read '''pfSense''' and FreeBSD forums to learn about the issues from people who use Realtek network interface cards. Sometimes you’ll get something working, but often you will get headaches and nightmares that are not worth the cost savings. Realtek network cards are best avoided in pfSense &amp; similar setups due to known issues with poor performance &amp; compatibility. Intel network interface cards are preferable for reliability &amp; better support in open-source projects like pfSense.

<blockquote>'''Note of Appreciation''': pfSense developers have created drivers for network interface chipsets like the 225 ([https://github.com/freebsd/freebsd-src/commit/517904de5ccac643589c71ac0d2751797f89e4f9 citation 1], [https://github.com/pfsense/FreeBSD-src/commit/9ffb4c0adab4853ab752ecda6a5ff59ea943af4e citation 2]) that didn’t exist before. Intel network interface cards are known to have better performance &amp; reliability in FreeBSD systems than Realtek chipsets. The ecosystem of open source firewalls are invested in providing support for these chipsets, providing solutions when the manufacturer doesn’t.

This is an excellent argument in favor of [https://www.reddit.com/r/PFSENSE/comments/uuigfy/is_the_intel_i225v_nic_ok/ paying money for open source software]. The igc driver for the i225 Intel network chip was made available to everyone! Commercial users, non-paying users of pfSense, and other FreeBSD based routers/firewalls all benefit from people paying for open source software. Top notch programmers wrote these drivers because they were able to pay their rent &amp; bills doing so.

When you pay for open source software, you are sending a message that it makes sense for top notch programmers to spend money developing open source code that doesn’t abuse you rather than going to work for facebook.
</blockquote>
<ol start="2" style="list-style-type: decimal;">
<li>'''Buy from Reputable Vendors:''' Avoid counterfeit products by purchasing from trusted sellers. There are many counterfeit cards out there.</li></ol>

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:lu55028jxaty_tmp_42645421.png
File:lu55028jxaty_tmp_3d7f0c6c.png
</gallery>

<ol start="3" style="list-style-type: decimal;">
<li>'''Vendors don’t know the difference:''' Many vendors selling knockoff cards do not even know they are doing it. Wholesale liquidators operate with low profit margins while selling a wide variety of equipment and lack the time and expertise to vet all of what they sell. As a result, many vendors sell counterfeit and fake Intel network cards.</li></ol>

* '''Recommended:''' ''The Art of Server'' on eBay ([https://www.ebay.com/str/theartofserver link])
* '''Example product:''' Intel X540 ([https://www.ebay.com/itm/166585171595 link])

'''Verify Compatibility:''' Make sure the card fits your PC’s available slots.

* Be wary of non-standard form factors or connectors.

'''HINT:''' Buying cards that are branded from server re-sellers is a good way to avoid fakes. For instance

<span id="dont-buy-digiorno"></span>

=== Don’t buy Digiorno ===

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:lu55028jxaty_tmp_19879e61.png
File:lu55028jxaty_tmp_be9591e5.png
File:lu55028jxaty_tmp_cb6dfb59.png
</gallery>

Buying used network cards, and used hardware, is ok. Actually, it’s encouraged; it’s a great way to buy better hardware than you’d otherwise be able to afford, and it avoids senseless waste. However, be careful to not buy Digiorno. There are amazing deals to be found in the used server world, but it is also a jungle ready to eat you alive if you’re naive enough to believe those crazy folks have any respect for the civilized world of standardized connectors.

'''Good vendors will be able to tell you the difference between normal hardware and Digiorno. If they do not know the difference, YOU DO NOT WANT TO BUY FROM THEM!'''

Building a DIY '''pfSense''' router with an old desktop PC and quality Intel NICs is likely to provide a more reliable and expandable solution than generic mini PCs. With a random mini PC, if you get a bad network interface card, you’re out of luck. With your old desktop PC, you can choose the network interface card. Want 2.5GbE? Get another card. Want 10 Gbps? Get another card. Want fiber? Get another card. Have a card with the wrong chipset? Swap in another card.

We are going down a 10+ hour rabbit hole of hell setting up all sorts of confusing, crazy GNU/Linux software. Even a 1% increase in the likelihood of this being more difficult as a result of random garbage Amazon hardware isn’t worth it to me for $100-$200 in savings.

I chose an Intel NUC because it has two quality NICs, and I was able to find one affordably. You do not have to buy the computer I bought to use as a router: this is your journey!

'''Note:''' There is no one “right” way to do this. As long as you use a stable, quality computer with GOOD network interface cards that the '''pfSense''' &amp; <code>FreeBSD</code> community approve of, you are set!

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:lu55028jxaty_tmp_62a48d1.png
File:lu55028jxaty_tmp_3759a8d9.png
File:lu55028jxaty_tmp_ac097d76.png
File:lu55028jxaty_tmp_1ae997fd.png
</gallery>

<span id="step-1-downloading-pfsense-and-preparing-a-bootable-usb-drive"></span>
== Step 1: Downloading pfSense and Preparing a Bootable USB Drive ==

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:lu55028jxaty_tmp_f0537346.png
File:lu55028jxaty_tmp_8ddd142e.png
File:lu55028jxaty_tmp_487b2ff3.png
</gallery>

<span id="download-pfsense"></span>
==== 1.1 Download pfSense ====

'''pfSense'''’s website has unfortunately become [https://www.pfsense.org/download/ cancer] in recent years. While I am all for paying for software, the concept of having to add to cart, checkout, and insert billing information to download a free image… no. Avoid using this version of the website. Instead, [https://atxfiles.netgate.com/mirror/downloads/ go here]. Feel free to buy it and pay for their support, but don’t jump through stupid hoops.

# Open your web browser and visit the [https://atxfiles.netgate.com/mirror/downloads/ pfSense mirror site].
# Choose the correct architecture for your system (usually <code>amd64</code> for most modern computers, including Intel NUCs). If you don’t know what the difference is between these, pick <code>amd64</code>.
# Select the USB installer image (.img.gz) from the available options.

<span id="unzip-the-downloaded-pfsense-file"></span>
==== 1.2 Unzip the Downloaded pfSense File ====

# After the download completes, you’ll need to uncompress (unzip) the file.
# The file typically ends with <code>.gz</code>. Use the right tool for your operating system:

<ul>
<li><p><code>Linux or macOS:</code> Open a terminal and run the following command:</p>
<pre>gzip -d pfSense-CE-memstick-*.img.gz</pre></li>
<li><p><code>Windows:</code> Use a tool like 7-Zip. Right-click the file, choose “Extract Here,” and let the tool unzip it.</p></li></ul>

<span id="create-a-bootable-usb-drive-with-the-pfsense-image"></span>

==== 1.3 Create a Bootable USB Drive with the pfSense Image ====

'''Warning:''' This process will erase everything on the USB drive.

# Insert a USB flash drive (at least 4GB in size) into your computer.
# Use one of the following methods to write the pfSense image to the USB drive:

<span id="windows"></span>
===== Windows: =====

# Download and install [https://en.wikipedia.org/wiki/Rufus_(software) Rufus].
# Open Rufus and select your USB drive.
# Click the ''“SELECT”'' button and choose the unzipped <code>.img</code> file you downloaded.
# Click ''“Start”'' and let Rufus create the bootable USB.

<span id="GNU/linuxmacos"></span>

===== GNU/Linux or macOS: =====

<ol style="list-style-type: decimal;">
<li><p>Open the terminal and type one of the following commands depending on the system used:</p>
<pre>sudo fdisk -l # GNU/Linux</pre>
<pre>diskutil list # macOS</pre></li>
<li><p>Make note of drives in the system. Do not erase these.</p></li>
<li><p>Plug in the flash drive.</p></li>
<li><p>Open the terminal and type one of the following command again:</p>
<pre>sudo fdisk -l # GNU/Linux</pre>
<pre>diskutil list # macOS</pre></li>
<li><p>Make note of the drive that was not present before. Write it down.</p></li>
<li><p>Double-check size/brand/model to make sure this new device is the device you plugged in.</p></li>
<li><p>Now, unplug the drive you just plugged in.</p></li>
<li><p>Run:</p>
<pre>sudo fdisk -l # GNU/Linux</pre>
<pre>diskutil list # macOS</pre></li>
<li><p>Does the drive you wrote down in step 5 still appear? If so, you made a mistake, and you’re on your way to deleting all of your data. Don’t do that. Do not pass go, do not collect $200 – back to the beginning. If not, you can now plug your drive back in.</p></li>
<li><p>Run:</p>
<pre>sudo fdisk -l # GNU/Linux</pre>
<pre>diskutil list # macOS</pre></li>
<li><p>If the drive that did not appear last time, appears this time, and is the same device as in step 5, you are likely on your way to not erasing your entire system. Good job, that makes you less of an idiot than me; a low bar, but it’s something.</p></li>
<li><p>Run the following, replacing <code>/dev/sdX</code> with your drive, and replace the pfSense img file with the filename of your image file:</p>
<pre>sudo dd if=pfSense-CE-memstick-serial-*.img of=/dev/sdX bs=1M status=progress</pre></li></ol>

Your bootable USB drive with pfSense is now ready for use! If you managed to erase your entire computer by writing pfSense’s image to your operating system drive EVEN AFTER all of this, congratulations, you’re almost as stupid as me.

<span id="step-2-disable-secure-boot-and-install-pfsense-on-the-intel-nuc"></span>

== Step 2: Disable Secure Boot and Install pfSense on the Intel NUC ==

Before you can install pfSense, you’ll need to disable Secure Boot if you are using a modern computer. Many modern computers, especially those pre-installed with Windows 10 or 11, come with Secure Boot enabled, preventing you from booting into an operating system that isn’t signed by Microsoft initially. Since pfSense is open-source and unsigned, we need to disable Secure Boot to start our installation.

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:lu55028jxaty_tmp_827ab6bb.png
File:lu55028jxaty_tmp_582a2319.png
</gallery>

<span id="disabling-secure-boot-in-bios"></span>
=== 1. Disabling Secure Boot in BIOS ===

# '''Insert the USB Drive'''
#* Plug in the USB drive containing the pfSense installation image into one of the USB ports on your Intel NUC.
#* Make sure this is done before you power on the device.
# '''Enter the BIOS'''
#* Power on the Intel NUC and immediately start pressing F2 (or the designated key for your system) to access the BIOS settings.
#* Keep pressing this key until you enter the BIOS. On some systems, the BIOS key may be different (e.g., Delete or Esc), but F2 is common for most systems.
# '''Disable Secure Boot'''
#* Inside the BIOS, navigate to the Boot section.
#* Locate Secure Boot and toggle it to Off. Depending on your BIOS, Secure Boot may be located under the Security or Boot sections.
#* Once Secure Boot is disabled, you’re ready to install pfSense.
# '''Set Boot Priority'''
#* In the BIOS, go to Boot Priority settings.
#* Set your USB drive as the first boot device. This will allow the system to automatically boot from the USB drive containing the pfSense installer.
#* Alternatively, you can press <code>F12</code> (or the appropriate key) during boot to manually enter the boot menu &amp; select the USB drive each time.
# '''Save and Exit BIOS'''
#* Press <code>F10</code> to save your changes and exit the BIOS, or whatever key does it on your machine.
#* The system will now reboot, and if the USB drive is set as the first boot option, it should boot directly from the USB flash drive and load the pfSense installer.

<span id="step-3-installing-pfsense-on-the-intel-nuc"></span>
== Step 3: Installing pfSense on the Intel NUC ==

<span id="boot-from-the-usb-flash-drive"></span>
=== Boot from the USB Flash Drive ===

<span id="power-on-the-intel-nuc"></span>
==== 1.1 Power on the Intel NUC ====

* Make sure the USB drive containing the pfSense installer is still plugged into the Intel NUC.
* Power on the NUC and press <code>F10</code> (or the relevant boot menu key) to select the USB drive as the boot device.

<span id="select-the-usb-drive-in-boot-menu"></span>
==== 1.2 Select the USB Drive in Boot Menu ====

* In the boot menu, you’ll see a list of available boot devices. Select the USB flash drive that contains the pfSense installer.
* Press <code>Enter</code> to boot from the USB drive.

<span id="begin-the-pfsense-installation"></span>
=== Begin the pfSense Installation ===

<span id="pfsense-installer-menu"></span>
==== 2.1 pfSense Installer Menu ====

* After a few moments, the pfSense installer menu will appear.
* Use the arrow keys on your keyboard to select '''Install''' and press <code>Enter</code> to begin the installation.

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:lu55028jxaty_tmp_8890b270.png
</gallery>

<span id="choose-installation-method"></span>
==== 2.2 Choose Installation Method ====

* The installer will guide you through the process. When prompted to choose an install method, select '''Auto (ZFS)''' for the file system.
* ZFS is a great file system that offers data integrity, snapshots, and other advanced features. You probably won’t use most of them, but it’s still an excellent choice.

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:lu55028jxaty_tmp_41fc31d.png
File:lu55028jxaty_tmp_e354a54e.png
</gallery>

<span id="select-the-correct-installation-drive"></span>
=== Select the Correct Installation Drive ===

Raidz1 is a good option in that it allows one of the drives in your machine to die, and the router to keep going. This requires you have not one, but two drives inside your router machine. This is not a bad idea. You should be making a backup file of your router anyway so that you can restore regardless of what happens to any and all of the hardware on this one: but, this will allow the router to keep working even if a single drive dies. I am using stripe, no redundancy, which is the option you will be picking if you have only one drive in the router.

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:lu55028jxaty_tmp_bcad25ab.png
</gallery>

<span id="select-internal-ssd-or-hard-drive"></span>
==== 3.1 Select Internal SSD or Hard Drive ====

* The next step is to select the disk where '''pfSense''' will be installed. This is a very important step, so pay close attention.
* You will see a list of drives. The USB drive will usually appear as a small capacity device (e.g., 4GB or 8GB).
* Choose the larger drive that represents your Intel NUC’s internal SSD or hard drive (e.g., 256GB, 512GB).
* '''Important:''' “generic-mass-storage-class” is usually your external USB flash drive. '''If you’re using a PC with an internal drive, there’s a 99% chance that “generic-mass-storage-class” is NOT what you want to select unless you’re intentionally installing to a USB mass storage device (which is not recommended for a permanent installation).'''
* In my case, the Micron SSD was my internal SSD. Your drive name may be different, but look for a larger capacity drive that matches what you know is inside your NUC or PC.
* Use the arrow keys to highlight the correct drive, press space to select the drive, then press Enter to confirm your selection.

<span id="confirm-erase-and-installation"></span>

==== 3.2 Confirm Erase and Installation ====

* Once the correct internal drive is selected, the installer will ask if you want to erase the drive and proceed with the installation.
* This will erase all data on the selected drive. Make sure you’ve backed up any important data before proceeding.
* Confirm by selecting '''Yes'''. The installer will now copy files and set up '''pfSense''' on the internal drive. This may take a few minutes.

<span id="complete-the-installation-and-reboot"></span>
=== Complete the Installation and Reboot ===

<span id="remove-the-usb-flash-drive"></span>
==== 4.1 Remove the USB Flash Drive ====

* After the installation is complete, you’ll be prompted to reboot the system.
* Before rebooting, remove the USB flash drive from the Intel NUC. This makes sure it boots from the newly installed '''pfSense''' system on your internal drive.

<span id="reboot-and-load-pfsense"></span>
==== 4.2 Reboot and Load pfSense ====

* After removing the USB drive, press <code>Enter</code> to reboot the system.
* The Intel NUC will now boot into '''pfSense''' from the internal drive, and you’ll be greeted with the '''pfSense''' console screen.

Now that '''pfSense''' is installed, you’re ready to proceed with the initial configuration. This includes setting up your WAN (external network) and LAN (internal network) interfaces to make the NUC function as your network router.

<span id="step-4-first-time-configuration-of-pfsense"></span>
== Step 4: First-Time Configuration of pfSense ==

Now that you have pfSense installed on your device, it’s time to set it up and configure the basic settings. This step will cover configuring the '''WAN''' (internet) and '''LAN''' (local network) interfaces, setting IP addresses, and making sure everything is ready for further setup.

<span id="connecting-and-booting-up-pfsense"></span>
=== 1. Connecting and Booting Up pfSense ===

<span id="connect-your-devices"></span>
==== 1.1 Connect Your Devices: ====

* Plug your cable modem into one of the Ethernet ports on your pfSense device.
* Plug your desktop computer (the one you’re using to set everything up) into the other Ethernet port.
* At this point, you don’t need more than these two connections.

<span id="power-on-and-watch-the-boot-process"></span>
==== 1.2 Power On and Watch the Boot Process: ====

* Turn on your pfSense device.
* You’ll see a lot of text scrolling on the screen as the system boots up. Don’t worry if it seems overwhelming—this is normal.
* Pay close attention to the information displayed, especially towards the end of the boot process. Look for any text related to an '''IP address''' or '''interface name''', like what is pictured below:

<blockquote>'''NOTE''': Interface names can be ascertained by looking at what is going on as the machine boots. This is helpful for later! Refer to images below.
</blockquote>
<span id="initial-configuration-steps"></span>
=== 2. Initial Configuration Steps ===

<span id="vlan-setup-prompt"></span>
==== 2.1: VLAN Setup Prompt ====

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:lu55028jxaty_tmp_4cd80f0d.png
</gallery> - One of the first prompts you’ll see is: '''“Should VLANs be set up now?”'''

<span id="vlan-setup-prompt-1"></span>
==== 2.1: VLAN Setup Prompt ====

* What is a VLAN? VLAN stands for '''Virtual Local Area Network'''. It’s a way to create separate networks within your network. For example, if you have a switch with 52 ports and want to have five different networks all connected to your router with just one cable, you’d use VLANs. However, this is way too advanced for what we’re doing here.
* You may see a bunch of random text appear before you have a chance to respond. Don’t worry, you haven’t missed your opportunity to input. You can still type ‘n’ and hit enter when you’re ready.
* This is just normal open-source nerd UI/UX that is not designed for normal people. You will see a lot of this. That is why we’re here!
* For now, press '''‘N’''' to skip VLAN setup. We’re setting up just one local network, so VLANs aren’t necessary at this stage. You may do this later with the wifi section to have segmented wifi networks for trusted &amp; untrusted devices &amp; to limit their access, '''but that does not have to be done right now and can be done later!'''

<span id="wan-and-lan-interface-assignment"></span>
==== 2.2: WAN and LAN Interface Assignment ====

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:lu55028jxaty_tmp_78cbbbb8.png
</gallery>

* Next, pfSense will show you which interfaces are available on your device. This is where you assign the Ethernet ports for '''WAN''' (internet) and '''LAN''' (internal network).
* Pay close attention to the bottom third of the screen. You’ll see information about which interface (e.g., <code>em0</code> or <code>igb0</code>) has received an IP address. The interface that received an IP address is most likely your '''WAN interface'''. In my case, <code>em0</code> is the interface attached to Spectrum cable internet; makes sense that it’s sad…
* Your desktop PC is not going to “provide” an IP address to the router; it is going to try to '''retrieve''' an IP address from the router. This is how we determine that the interface that has received an IP address is the WAN interface connected to our modem.
* The names of these interfaces may vary depending on your hardware and pfSense version. Don’t worry if they don’t match exactly what you see in this guide.

When prompted:

# '''Enter WAN Interface Name:'''
#* Input the name of the interface that received an IP address (e.g., <code>em0</code>).
# '''Enter LAN Interface Name:'''
#* Input the name of the other interface (e.g., <code>igb0</code>).

Confirm the interface assignments when prompted. This tells '''pfSense''' which port to use for '''WAN''' (internet) and which for '''LAN''' (local network).

<blockquote>'''NOTE''': This is the IP address that you would be accessing the '''pfSense''' web interface on. This is also your “gateway” address, i.e., what your computer connects to in order to get an IP address, and before it connects to any IP outside of this subnet (subnet = other devices on your LAN, e.g., cellphone, TV, file server, etc.).
</blockquote>
<span id="configuring-lan-ip-address"></span>
=== 3. Configuring LAN IP Address ===

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:lu55028jxaty_tmp_ebc6f9c0.png
</gallery>

<span id="default-lan-ip"></span>
==== 3.1: Default LAN IP ====

After assigning interfaces, '''pfSense''' will show you the default LAN IP address, usually '''192.168.1.1'''.

This is the IP address of your router ('''pfSense''') within your local network.

Any device that connects to the router will be assigned an IP address in the '''192.168.1.x''' range by default. For instance, your PC may grab an IP of 192.168.1.46, 192.168.1.16, etc., if set to connect automatically via <code>DHCP</code> (Dynamic Host Configuration Protocol).

<code>DHCP</code> means when you connect to a router it grabs an IP address/DNS server/etc. to you by default, “Plug N Play” style. This is the default configuration of most devices you will ever connect to the internet unless you went out of your way to re-configure them. This includes your computer, cellphone, game console, IoT devices, security cameras, etc. They’re all connecting via <code>DHCP</code>.

<span id="changing-the-lan-ip-optional"></span>
==== 3.2: Changing the LAN IP (Optional) ====

Requirements:

You don’t need to change this unless you have a specific reason to do so, such as conflicts with other networks you’re using. I have chosen to change it, and will be working with the following configuration throughout this guide. '''You do not have to follow what I am doing, but if you want to be able to copy &amp; paste along with me addresses of things, feel free to do it this way, it won’t hurt.'''

# '''Set Interface IP address'''
#* The number for the LAN interface was <code>2</code> in my case
# '''Configure the new LAN IPv4 address via DHCP'''
#* Choose <code>n</code>
#* This isn’t referring to having DHCP so that clients who connect can get an IP address. This means should this interface have a dynamic IP itself, meaning the the router/gateway would have a different IP each time we connect to it. There is no need for this.
# '''Enter the new LAN IPv4 address'''
#* <code>192.168.5.1</code> is my LAN IPv4 address that I will choose for my router.
#* This is where your '''pfSense''' router will be accessible via web browser. This will be your gateway address, and this will be your DNS server.
# '''Enter LAN IPv4 subnet bit count'''
#* <code>24</code> is the subnet bit count
#* (This is shorthand for a subnet mask of <code>255.255.255.0</code>).
# '''IPv4 upstream gateway address'''
#* Press enter for none.
# '''Configure IPv6 address for LAN interface via DHCP6'''
#* Press <code>y</code> , we’re not using IPv6 in this guide anyway.
#* I hit <code>y</code>, you can hit <code>n</code> and specify an address manually, but I will not be using IPv6 so it makes no difference to me, no need to specify an address I have to remember for something I will never use.
#* You’re welcome to set up an IPv6 home network if you want; I am not covering that here.

<span id="dhcp-setup"></span>
==== 3.3: DHCP Setup ====

# '''DHCP (Dynamic Host Configuration Protocol)''' automatically assigns IP addresses to devices on your network. This makes it easier to connect new devices without manually configuring IP settings on each one. This is what allows clients to be able to get an IP address automatically as soon as they connect via Wi-Fi or with an ethernet cord into your switch. You want this so that by default people can go online without having to specify their IP manually.
# When asked if you want to configure DHCP, choose '''Yes'''.
# Set the DHCP range. This is the range of IP addresses that will be assigned to devices on your network. For example:
#* '''Start Address:''' <code>192.168.5.2</code>
#* '''End Address:''' <code>192.168.5.254</code>
# Since we have our router on <code>192.168.5.1</code>, the next address that’s available is <code>192.168.5.2</code> which is the start, and <code>192.168.5.254</code> as the end.
# For ''Do you want to revert to HTTP as the webconfigurator protocol'', choose <code>n</code>. No need to use HTTP instead of HTTPS. We’re never going to connect to this without a VPN anyway, so HTTP vs HTTPS isn’t the biggest security deal in the world, but it’s a good practice to use HTTPS whenever possible.

This allows up to 254 devices on your local network, which is more than enough for most home setups. If you have more than 254 devices at home, you’re likely not reading a beginner’s guide from a board repair person cosplaying as a sysadmin.

If you want to go crazy, you can do a different setup entirely: change the LAN IP to something even less common if you want to avoid conflicts, such as <code>172.16.10.1</code> as a LAN IP, subnet 24. This would allow 254 devices that would be given IPs such as <code>172.16.10.2</code>, <code>172.16.10.30</code>, etc.—and your '''pfSense''' router web interface would be accessible on <code>172.16.10.1</code>. When you connect to other people’s networks, if you don’t disable LAN access in the OpenVPN android client, and their network has a <code>192.168.1.1</code>, and yours has a <code>192.168.1.1</code>… You see where this is going. Chances are they don’t have a <code>192.168.5.1</code> though.

<blockquote>'''NOTE:''' If both your home network and a remote network you’re connecting from via VPN use the same IP range, you can end up with routing &amp; connectivity issues. Let’s say you’re at a coffee shop. You connect via wifi. On their network, you are 192.168.1.3. You connect to your home network via your VPN, and you want to connect to your local mailserver… but you both have the same pos linksys wrt54g router, which defaults everyone to 192.168.1.*. so you try to connect to 192.168.1.3. Do you see where this is going?

Changing your home network to a less common IP range can mitigate this risk. Always check the IP range of networks you frequently connect to and adjust your home network accordingly. Or, just make yours some weird-ass number that nobody else will be using. The latter works for me.
</blockquote>
<span id="finishing-up"></span>

=== 4. Finishing Up ===

At this point, the basic configuration is complete. You can now:

# Unplug the monitor, keyboard, and mouse from your '''pfSense''' device.
# Put away your keyboard and mouse.
# Turn your cable modem off for a minute or two, and then plug it back in. Some modems get mad when you plug in a new router.

<blockquote>'''NOTE:''' Configuring the LAN IPv4 address and subnet mask sounds confusing if you’re used to plugging in your 50 year old Linksys WRT54G &amp; getting going. It’ll get easier with time, but for now, let’s go over what some of these pieces do. You can always come back to this later.

'''What is the LAN IPv4 Address?'''
The LAN IPv4 address is the IP address assigned to your router on your local network. All your devices from your computer, phone, or smart TV ''(if you are reading this and still using a smart tv…)'' use that address as the “gateway” to get to the internet &amp; also to communicate with each other. The default configuration is that pfSense assigns <code>192.168.1.1</code> as the LAN IP address. This is the norm for most routers.

* This address is special because it tells devices where to send data when they want to leave your network. For example, if your PC needs to visit <code>apple.com</code>, it sends the request to the router’s LAN IP (<code>192.168.1.1</code>, otherwise known as the gateway), which then forwards it to the internet.

* If you’re not changing anything, you can stick with the default (<code>192.168.1.1</code>). I change it because everyone uses <code>192.168.1.1</code>. If you use a VPN or other networks frequently, changing it to something like <code>192.168.5.1</code> can avoid headaches down the line. If I am trying to connect to <code>192.168.1.1</code> on my home network, but <code>192.168.1.1</code> is the gateway IP of the wifi router my phone is connected to at my friend’s house… you see where this gets confusing.

'''What is a Subnet Mask?'''
A subnet mask is what defines the “size” of your local network. Your LAN is like a neighborhood; the subnet mask is like a property line that goes over how many houses can fit in the neighborhood.

* The default subnet mask for most home networks is <code>255.255.255.0</code>. This tells your router that there can be up to 254 devices (playstations, phones, computers, etc) connected to your network. That’s a lot. If you have more than 254 devices in your house, you’re probably not reading this guide.
* This subnet mask is written abbreviated as <code>/24</code> because the first 24 bits (the <code>255.255.255</code> part) of the address are fixed while it’s only the last 8 bits are available for device addresses.

'''Why Configure a Static LAN IP?'''
When you assign a static LAN IP to your router, you’re making sure that its address never changes. It would make no sense to have a router IP that changes constantly. Your servers &amp; devices all need to connect to the router, so keep the router where it is. Moving it around senselessly makes no sense. It would be akin to Walmart changing its address every day.

* Imagine your router’s address was constantly changing. One moment it’s at <code>192.168.1.1</code>, and the next, it’s at <code>192.168.1.87</code>. Your devices would be as confused as I am when I call a [https://www.youtube.com/watch?v=qFVwQCFhKSE New York state tax office].

* By giving a static IP like <code>192.168.5.1</code> to the router, I’m making sure that everything in your network knows where to go.

'''Step-by-Step explanation if you’re still confused:'''

'''Set Interface IP Address:''' - When it asks you to “Set interface IP address,” this is where you’re assigning the LAN IPv4 address. Think of it as giving your router its permanent address in your local network. Enter <code>2</code> to configure the LAN interface.

'''Configure the New LAN IPv4 Address:''' - Here, you’re telling '''pfSense''' what address you want to use for the router. For example, <code>192.168.5.1</code> makes your router accessible at that address.
- Remember: This is the gateway address that all your devices will use to connect to the internet. Write it down somewhere because you’ll need it later to log in to the '''pfSense''' web interface.

'''Enter LAN IPv4 Subnet Bit Count:''' - This is where you specify the subnet mask abbreviated. For most home setups, the bit count is <code>24</code>, aka <code>255.255.255.0</code>. This allows up to 254 devices to connect to your network. If you’re just starting out, stick with <code>/24</code>. - '''To keep it simple when you see <code>192.168.5.0/24</code> what they mean is everything from <code>192.168.5.1</code> to <code>192.168.5.254</code>. ''' - ''Why not use a bigger subnet?'' Because you’re reading a beginner’s guide. How about you get one device to work in your broom closet before going for over 254?

'''IPv4 Upstream Gateway Address:''' - This is asking if your LAN interface needs a separate gateway to reach the internet. Since your router '''is''' the gateway for your LAN, just press Enter to leave this blank. - '''Your LAN doesn’t need to forward traffic anywhere else because the router handles it.'''

'''Configure IPv6 Address for LAN Interface via DHCP6:''' - You’re not using IPv6. Forget about IPv6 for now. We’ll get to how this makes using your VPN a nightmare later on. If you are not a datacenter or a sysadmin for amazon web services, you have no need for ipv6 in your life at this stage.
</blockquote>
<span id="accessing-the-pfsense-web-interface"></span>

=== 5. Accessing the pfSense Web Interface ===

Now that the basic network setup is complete, you can access the '''pfSense''' web interface to configure more advanced settings.

# On your desktop computer (connected to the LAN port), open a web browser.
# Go to <code>https://192.168.5.1</code> or <code>https://pfSense.home.arpa</code>.
# You may see a security warning in your browser. This is because '''pfSense''' is using a self-signed SSL certificate, which is fine for local networks. Click '''“Advanced”''' and proceed to the site.
# Log in with the default credentials:
#* '''Username:''' <code>admin</code>
#* '''Password:''' '''pfsense'''
# Once logged in, you’ll be prompted to change the default password. Set a strong password to secure your router.

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:lu55028jxaty_tmp_d07f1499.png
File:lu55028jxaty_tmp_d9cfd77d.png
File:lu55028jxaty_tmp_e583100f.png
File:lu55028jxaty_tmp_6d87d663.png
File:lu55028jxaty_tmp_c9bf064a.png
</gallery>

<span id="initial-web-setup-wizard"></span>
==== 5.1: Initial Web Setup Wizard ====

# '''Set the Hostname:'''

* Choose a hostname for your '''pfSense''' router. This can be something simple like “pfsense” or “home-router.” You will be able to access the router at <code>pfsense.home.arpa</code> once we set everything up with DNS later, instead of having to visit the router’s web interface based on its IP address. If you typed <code>roflcopter</code> into this box, you would be able to access your router at <code>https://roflcopter.home.arpa</code> rather than typing in [https://192.168.5.1/ https://192.168.5.1] – you get the idea.

<ol start="2" style="list-style-type: decimal;">
<li>'''Set DNS Servers:'''</li></ol>

* For now, you can use a public DNS provider like '''Google DNS (8.8.8.8)''', but we’ll replace this with AdGuard DNS or similar later for ad-blocking.
* Uncheck the option to '''“Allow DNS server list to be overridden by DHCP/PPP on WAN,”''' so your ISP cannot override the DNS settings you choose.

<ol start="3" style="list-style-type: decimal;">
<li>'''Time Zone:'''</li></ol>

* Set the correct time zone for your location (e.g., '''US Central''' if you’re in Texas).

<ol start="4" style="list-style-type: decimal;">
<li>'''Final Steps:'''</li></ol>

* Once these settings are configured, hit '''“Next.”''' It’ll ask you to configure the WAN interface. Unless you have a funky setup, you need not change anything here. This is not for you to mess with.
* It’ll ask you to configure the LAN interface again, but you need not touch anything, remember we already did this and the settings you put in earlier should be what shows up.
* It’ll ask you to make a secure password; it is a good idea to set a secure password and save it in a password manager. No post-it note on the monitor nonsense!
* You’ll be taken to the final page where you can apply the settings and restart the web interface.

<span id="final-check-and-preparing-for-the-next-steps"></span>

=== 6. Final Check and Preparing for the Next Steps ===

At this point, '''pfSense''' is fully installed, and the basic configuration is complete. Here are some final steps and checks:

# It’s a good idea to restart your cable modem when you make these changes, especially if it was previously connected to another router.
# You might want to reset the internet connection on the device you’re using to access the '''pfSense''' web interface, especially if it was connected to a different network before.
# Before we move forward to setting up additional features (like ad-blocking), make sure your internet connection is stable and working as expected.
# Test your internet connection by browsing the web from a device connected to the LAN.
# Remember, you can now manage everything through the web interface. You shouldn’t need to directly connect to the '''pfSense''' device with a monitor and keyboard again unless something breaks. Put the keyboard, mouse, and monitor plugged into that '''pfSense''' device away; we’re (hopefully) never touching that again. '''If you are, that means something bad has occurred.'''
# If you encounter any issues, re-check everything you did.

'''Congratulations!''' Your '''pfSense''' router is now set up and ready for use. Now the real fun begins. :)

<span id="setting-up-freedns-for-dynamic-dns"></span>

= '''Setting Up FreeDNS for Dynamic DNS''' =

<span id="why-do-you-need-dynamic-dns"></span>
== Why Do You Need Dynamic DNS? ==

'''Your IP address changes.'''

Your IP address is like your home address or phone number. You want this to be static - as in, doesn’t change. Imagine if all of the road names and highway exits changed each day, or if your friend’s phone number changed every day. This would be a mess. How would you know who to call? It would be very confusing. This is how it is when you have a dynamic IP.

Most of you setting up a home server likely have a residential internet plan from providers like Spectrum, AT&amp;T, or Verizon. Unlike professional hosting services with static IPs, residential plans assign dynamic IP addresses that change as often as the relationship partners of people with borderline personality disorder. This can be a problem when you want to access your home network remotely.

What if you had a speed dial button that automatically kept track of that friend’s changing number, and just allowed you to reach your friend every time you pressed on their name? That’s how a dynamic DNS works.

Even if you DO manage to memorize 33.84.182.1, imagine having to memorize a new number every week. Or ''every day!''

And what if it changes in the ''middle of the day?'' Imagine having to check your IP address every day, or calling home &amp; going ''“hey honey, can you go to whatismyip.com and give me the number so I can add something to my calendar? Thanks!”''

That would be horrible.

What you want to do is go to ''chrisserver.mooo.com'' or ''mysite.ddns.net'' and it takes you right to your server, every time. This is possible because someone else can do the work of keeping track of your router’s IP address and assigning it to that domain name. Or, something. That thing is a '''dynamic DNS provider.'''

This is where '''Dynamic DNS''' comes in handy. It automatically updates a friendly hostname to point to your current IP address, so you can always access your home network using a consistent address.

<span id="setting-up-freedns"></span>

== Setting Up FreeDNS ==

<span id="step-1-register-on-freedns"></span>
=== Step 1: Register on FreeDNS ===

We’re going to use a service called '''FreeDNS'''. It’s free, easy to use, and even has some fun domain options.

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:lu55028jxaty_tmp_4729b2e6.png
File:lu55028jxaty_tmp_45e15cd2.png
File:lu55028jxaty_tmp_7c63c31c.png
File:lu55028jxaty_tmp_9cd59f27.png
File:lu55028jxaty_tmp_684ec372.png
</gallery>

<span id="create-a-freedns-account"></span>
==== 1.1 Create a FreeDNS account ====

# '''Visit FreeDNS:''' Go to [http://freedns.afraid.org/ freedns.afraid.org].
# '''Register:''' Click on “Sign up Free” in the lower center of the page.
# '''Fill out form:''' Fill in the required fields (username, password, and email) and click “Create Account”.
# '''Verify your account''' by clicking the link in the confirmation email.

<span id="log-into-freedns-create-subdomain"></span>
==== 1.2 Log into FreeDNS &amp; create subdomain ====

Continue with the steps to set up your subdomain as needed.

This is going to be the “website name” we associate with our home server internet connection. When you visit <code>rossmanngroup.com</code>, this actually means <code>208.113.140.53</code>. When you type ''[http://rossmanngroup.com/ http://rossmanngroup.com]'' in your browser, you’re asking your browser to go to <code>208.113.140.53</code> and knock on port 80 to serve us a website. When you type <code>https://rossmanngroup.com</code> in your browser, you’re saying we’re going to <code>208.113.140.53</code> and knocking on port 443 to be served a website with https/ssl.

The '''subdomain''' enclosed in red in the screenshot above is the '''first part''' of the website name, and the '''domain''' enclosed in green is the '''second part''' of the website name. The '''destination''' enclosed in blue is where our combined website name leads us. So, <code>louishomeserver.chickenkiller.com</code> in the configuration above, would lead us to <code>8.8.8.8</code>

# After clicking the activation link from the FreeDNS email, you should be immediately logged in. You should save the username &amp; password they gave you in a password manager.
# '''Add a New Subdomain:''' Once logged in, click on “Add a subdomain” in the middle of the screen from the main menu. Or, click Subdomains on the left side menu.
# Fill out the fields:
## '''Subdomain:''' Choose a custom name (e.g., “louishomeserver”). That’s the part I circled in red in my screenshot above.
## '''Domain:''' Select one of the available free domains (e.g., <code>chickenkiller.com</code>). This is the green field in my screenshot above. '''You can get your own pretty, custom named .com address, but you’ll have to pay for it.'''
## '''Destination:''' Here’s the trick - put in a WRONG IP address on purpose (e.g., <code>8.8.8.8</code>). This will help us confirm if our setup is working later.
# The entire point of this is for our router to constantly ''update'' FreeDNS by telling it what our IP address is. If we put what our IP address is ''RIGHT NOW'' in this field, we won’t know for sure if '''pfSense''' is working properly with FreeDNS. We’d have to debug it through log files. Ew.
# Click “Save” to create your hostname.

<blockquote>'''NOTE:''' Setting an incorrect initial IP address lets us test that pfSense is correctly updating the dynamic DNS entry. This diagnostic step is an important one; screw things up &amp; make sure that the system you put into place to auto-fix-it fixes it. This is far less dangerous than the alternative, which is “assuming that it works.”
</blockquote>
<span id="get-the-update-url-from-freedns"></span>
==== 1.3 Get the update URL from FreeDNS ====

The update URL is the URL '''pfSense''' will access to tell FreeDNS that your domain name’s IP address has changed &amp; should change to the IP that your router is accessing that FreeDNS URL from.

# After saving, click “Dynamic DNS” from the upper left menu of choices.
# You’ll see your new subdomain at the bottom.
# Right-click on the “Direct URL” link next to your hostname and copy the link address.
# This URL is how we will update our IP address automatically. '''DO NOT SHARE THIS WITH ANYONE OR THEY WILL BE ABLE TO MESS WITH YOU ENDLESSLY BY CHANGING THE IP THAT YOUR NEW WEBSITE NAME ATTACHES TO AWAY FROM YOUR SERVER!!'''

<blockquote>'''NOTE:''' The Direct URL contains what is like a “password” necessary for updating your dynamic DNS record. Keep this URL secure and don’t share it publicly unless you want your dynamic dns domain name redirecting to goatse
</blockquote>
<span id="leave-freedns-page-open-make-sure-it-has-the-wrong-ip-for-you."></span>

==== 1.4 Leave FreeDNS page open &amp; make sure it has the WRONG IP for you. ====

# Either the ''[https://freedns.afraid.org/dynamic/ dynamic DNS page on FreeDNS]'' OR the ''[https://freedns.afraid.org/subdomain/ subdomains page on FreeDNS]''. Make sure the IP address is as we entered before, which is <code>8.8.8.8</code>.
# '''IT IS IMPORTANT THAT THIS IP ADDRESS NOT BE YOUR IP ADDRESS! WE WANT IT TO BE WRONG!'''
# Make sure it is still set to the <code>8.8.8.8</code> I told you to set it to before.
# If it is not, set it to <code>8.8.8.8</code>.
# Reload both pages. Still <code>8.8.8.8</code>? Good.
# We want this to be wrong – it changing from “wrong” to “not wrong” when we finish our work will mean that our setup works.

'''Why are we doing things this way?''' It takes an insignificant amount of extra time to do things like this, as a check against everything; from software glitches to my own carelessness and absent-mindedness.

Starting with the assumption that nothing works properly allows us to figure out at the very beginning if everything has been configured properly. Starting with the assumption that nothing works will allow us to figure out if our system ACTUALLY works BEFORE WE NEED IT TO WORK!!

You’ll see when we are testing certain features and functionalities and software like syncthing later that this comes in very handy. In the video, a connection will work &amp; have a green checkbox the first time, but local discovery will not work the second time, even though the server &amp; client IPs remain unchanged.

NEVER ASSUME A FIX WORKS WITHOUT BREAKING THE THING IT IS SUPPOSED TO FIX AND SEEING WHAT IT DOES.

Assume that nothing works, especially your own brain, and you will create systems that guard against much more than your own human error!

<span id="step-2-configuring-pfsense-for-dynamic-dns"></span>
== Step 2: Configuring pfSense for Dynamic DNS ==

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:lu55028jxaty_tmp_a6b5fa5b.png
File:lu55028jxaty_tmp_8b618542.png
File:lu55028jxaty_tmp_be361b57.png
File:lu55028jxaty_tmp_a9a56361.png
</gallery>

pfSense has to talk to FreeDNS regularly to tell it our IP.

<span id="log-into-pfsense"></span>
==== 2.1 Log into pfSense ====

Open the pfSense web interface and log in, at [https://192.168.5.1/ https://192.168.5.1] or [https://pfsense.home.arpa/ https://pfsense.home.arpa]

<span id="enter-dynamic-dns-settings"></span>
==== 2.2 Enter Dynamic DNS settings ====

In the pfSense dashboard, there is a menu on the top. Go to <code>Services &gt; Dynamic DNS</code>.

<span id="enter-dynamic-dns-entry"></span>
==== 2.3 Enter Dynamic DNS entry ====

# Click the “+ Add” button to add a new entry.
# '''Configure the Settings:'''
#* '''Service Type:''' Select “Custom” from the dropdown. This might seem counterintuitive since “freedns” exists as an option in this dropdown menu, but trust me here. You trust me… right? This is the green box in the screenshot I provided above.
#* '''Interface to Monitor:''' Select <code>WAN</code> (your external internet connection), this is the part circled in purple above.
# '''Interface to send update from:''' Select WAN, the part circled in purple above.
# '''HTTP API DNS Options:''' I check '''“Force IPv4 DNS resolution”''' because I have been scarred by my residential internet service provider’s issues with IPv6 before Spectrum bought Time Warner Cable. You don’t have to check this, but I check it because I hate IPv6 &amp; have it turned off entirely in my own setup. I shouldn’t be passing my prejudices onto my children. But here I am passing this one onto you :’(
# '''Update URL:''' Paste the Direct URL you copied from FreeDNS. Everything after the question mark in this URL is like your password and username combined.
    IF YOU ARE USING CLOUDFLARE, you need to use your Zone ID as the username and the token you just created (with the Zone.DNS - Edit permission) as the password. Otherwise, if you use the token alone, the status will always appear green, but you won’t be able to connect. You might end up spending four hours, like I did, debugging all sorts of issues until you finally find the answer in a four-year-old Reddit post. Also, make sure you disable proxy on cloudflare.
# '''Max Cache Age:''' When will this run? By default, this runs when an update is forced by you or when the router notices the WAN address (the IP address your ISP assigned to you) has changed. I see no harm in having it update once per day. If there’s any sort of stupid bug or issue or crap where it tried to update &amp; failed because the wind was blowing the wrong day, packet loss, etc… It costs literally nothing to do this, we’re in 2024, even people in the middle of nowhere have 768/128k DSL… There is zero downside to setting this to check at the minimum allowed interval, of once per day. ''“Inspect what you expect”'' as my stepmom would say, who was director of the Brookfield library. :) She could tell you firsthand that….. Nothing you expect people (OR COMPUTERS) to do, will they actually do.
# '''Description:''' Add something like “FreeDNS IP Update” to remember what this is for.
# '''Save the Configuration:''' Click “Save and force update” to store your Dynamic DNS settings.

'''If it went well''', the two areas I circled in red above should look similar to mine. A green checkmark under “status”, and the “cached IP” should be your actual IP address that you see when you go to a site like [https://www.whatismyip.com/ whatismyip.com]

<span id="step-3-see-if-dynamic-dns-actually-works"></span>

== Step 3: See if Dynamic DNS actually works ==

We purposely put an incorrect IP of <code>8.8.8.8</code> in there rather than our real IP address to make sure this actually works. Now we’re going to see what happens when we try to get it to work.

# Go to the pages I had you keep open before, the ''[https://freedns.afraid.org/dynamic/ dynamic DNS page on FreeDNS]'' OR the ''[https://freedns.afraid.org/subdomain/ subdomains page on FreeDNS]''.
# The IP was <code>8.8.8.8</code> before. Has it changed to the IP address that you see when you visit [https://whatismyip.com whatismyip.com], that is the WAN address in '''pfSense'''? If it is, you did good.
# '''Another way: Force an IP Change:'''
# Disconnect and reconnect your home internet connection to force your ISP to assign a new IP address.
## You can do this by rebooting your modem or temporarily disconnecting your internet connection.
## '''Sometimes, you may not be able to get a new IP, and that’s ok!'''
## Sometimes, you can’t get a new IP from your ISP immediately.
# As long as you input an incorrect IP address into the FreeDNS field for your subdomain when adding your subdomain to FreeDNS, and you saw it change to your WAN IP when you set up FreeDNS dynamic DNS in '''pfSense''', you are fine.

<span id="verify-dns-resolution"></span>
=== Verify DNS Resolution ===

To make sure your new hostname resolves to your home IP address, perform a DNS lookup from any device:

<ol style="list-style-type: decimal;">
<li><p>'''Open a Terminal or Command Prompt:'''</p>
<ul>
<li>On Linux or macOS, open Terminal.</li>
<li>On Windows, open Command Prompt.</li></ul>
</li>
<li><p>'''Run an <code>nslookup</code> Command:'''</p>
<pre>nslookup louishomeserver.chickenkiller.com</pre></li></ol>

Replace <code>louishomeserver.chickenkiller.com</code> with your actual hostname.

<ol start="3" style="list-style-type: decimal;">
<li><p>'''Verify the Result:'''</p>
<ul>
<li>The output should show your current public IP address associated with your hostname.</li>
<li>This confirms that your dynamic DNS is working correctly.</li>
<li>You could also just use <code>ping</code>.</li></ul>

<pre>ping louishomeserver.chickenkiller.com</pre></li></ol>

Does it ping your IP address? You’re good.

<span id="why-this-setup-is-important"></span>
=== Why This Setup Is Important ===

With this dynamic DNS setup, you no longer have to remember or manually track your public IP address, even when it changes. By using a hostname like <code>louishomeserver.chickenkiller.com</code> you can always access your home network remotely, no matter where you are or how often Spectrum goes down &amp; changes your IP on you.

This is useful for accessing home servers or services from outside your network via OpenVPN (next section!). '''pfSense''' Dynamic DNS service with FreeDNS makes sure that my chosen hostname always points to my current IP address. No matter how often my IP changes, I don’t have to change configuration settings in my programs.

<span id="openvpn-setting-up-secure-access-from-anywhere"></span>
= OpenVPN: Setting up Secure Access from Anywhere =

<span id="why-openvpn-why-do-i-need-this"></span>
== Why OpenVPN? Why do I need this? ==

Because opening ports for personal use is a bad idea!

“'''''but louis, every website &amp; hosting provider opens ports!”'''''

Webhosts and datacenters open ports so that millions of people can access their services. You’re opening ports to access a porn server in your closet. You’re not the same.

<span id="listing-the-ports-wed-have-to-open."></span>
=== Listing the ports we’d have to open. ===

Each one of these things needs its own open port on your router. That’s like having a house with 15 different doors, each one made of cardboard with a cutout in the middle allowing them to see in. No, we’re not doing that.

* '''Immich''' to do machine learning on your photos, because your self-image isn’t ''[https://imgur.com/a/HVr6oAz bad enough as it is]''.
* '''Home Assistant''' to pretend you’re Tony Stark
* '''Syncthing''' because [https://www.nytimes.com/2022/08/21/technology/google-surveillance-toddler-photo.html screw Google].
* '''MailCow''' because you think you can run email better than Google (if you’re reading this guide, you probably can’t)
* '''Frigate''' to catch your neighbor stealing your packages
* '''OnlyOffice''' because you’re too cheap for Microsoft 365
* '''FreePBX''' because… actually, I don’t know why you’d torture yourself with that. [https://www.youtube.com/watch?v=vWrkDOt_IfM&pp=ygUNbGVubnkgZnJlZXBieA%3D%3D Lenny] makes it worth it. Maybe

<span id="why-opening-every-port-is-dumber-than-an-820-2330-macbooks-hinge-design"></span>
=== Why Opening Every Port is Dumber Than an [https://rossmanngroup.com/unibody-macbook-pro-display-assembly-repair-replacement-service/ ''820-2330 Macbook’s hinge design''] ===

Here’s why exposing all of this directly is a terrible idea:

'''You’re Advertising What You’re Running''': Any script kid with a port scanner can see exactly what you’re running.

'''Your Software is Probably Full of Holes''': These projects are great, but they have 10,000 users, 5 of which believe they are entitled to 25 years of updates &amp; bugfixes after a $3 donation, maintained by [https://www.explainxkcd.com/wiki/index.php/2347:_Dependency one person in their spare time], whose users are assholes that think [https://www.reddit.com/r/immich/comments/1codh0p/comment/l5rfpu7/ feeding yourself off of your work is too much to ask for].

<div class="figure">

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:lu55028jxaty_tmp_37a2ee92.png
</gallery>

</div>
If I were smart (and evil), I could make a list of:

* Every IP address
* What software they ran
* What version they ran

Then, I’d keep up with exploits/vulnerabilities that are announced in the news. I’d go back to my list &amp; double check to see who’s running that software, and see if they work. At best, you become part of a botnet and waste some electricity mining my crypto. At worst, I’ve stolen all of your data &amp; use it to blackmail you.

I like these programs; they’re fun software! But, similar to my taste in relationships; it isn’t about '''who I''' '''''like.''''' It’s about '''who I''' '''''trust'''''. The software I have the most '''fun''' with isn’t who I’d trust with banking credentials (or my future children). Maybe I got that the wrong way around….

<span id="openvpn-only-1-port-to-open-with-better-security"></span>
=== OpenVPN: Only 1 Port to open, with better security: ===

'''One Port to Worry About''': Instead of 15 points of failure, we have one potential point of failure.

<blockquote>'''NOTE:''' OpenVPN uses a single port for all traffic, which is usually port 1194 UDP. Most OpenVPN servers will default to port 1194. Make sure your ISP didn’t block this. Bad ISPs will block ports commonly used for running servers so you pay 5x as much for the same internet unless you buy a “business”(extortion) plan. I paid $409.99/mo for 10 mbps upstream when I had a store in New York; hint, you’re not paying extra for better internet..
</blockquote>
'''Stealth Mode''': To the outside world, you’re just running OpenVPN. They can’t see your unpatched version of [https://github.com/pjenvey/hellanzb hellanzb] from 2007. ''(shout out to pjenvey if he’s reading this today!)''

<span id="openvpn-security-in-four-pictures"></span>
=== OpenVPN security in four pictures: ===

Here is what it’s like opening ports to a bunch of random open source projects people make in their spare time:

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:image55.jpg
</gallery> <gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:image56.jpg
</gallery>

Here is what it’s like only opening a port for OpenVPN.

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:image57.jpg
</gallery> <gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:image58.jpg
</gallery>

When you use OpenVPN, you are opening one port to get access to your network, with a door that many commercial interests have a stake in keeping strong. When you open ports for random crap, you have windows people can look through, and doors that look like… Well… Yeah. And 2 guys watching them.

<span id="decreasing-attack-surface-with-openvpn-is-a-best-practice"></span>
=== Decreasing Attack Surface with OpenVPN is a best practice ===

OpenVPN isn’t a hobby project coded by your cousin’s methhead roommate. This is used by everyone, from companies with more money than sense to just about anyone who doesn’t want their data plastered all over the internet:

* Having '''''ONE''''' service open to the public rather than 10 means a smaller attack surface.
* Having one service
* OpenVPN is designed with one purpose in mind, a secure connection.
* It is over 20 years old.
* Commercial interests (aka people actually paying money for software that rely on it for their infrastructure, not ''[https://www.reddit.com/r/immich/comments/1codh0p/comment/l5rfpu7/ this guy)]'' use &amp; rely on it.
* There are more eyes on the code of OpenVPN than <code>hellanzb</code>.

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:lu55028jxaty_tmp_f3e2603c.png
</gallery>

'''Marketing wankery? …Kind of, but they’re not lying here.'''

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:lu55028jxaty_tmp_29f791ff.png
</gallery>

'''Is this 100% accurate? No.''' Are more people for whom millions of dollars rides on the security of their software using OpenVPN than hellanzb. Yes!

Having a home server is cool. But the programs we’re talking about are used by [https://wiki.futo.org/index.php/FUTO:General_disclaimer  0.0001% of 0.000001%] of the world. OpenVPN can still have vulnerabilities; it isn’t perfect! But remember, in the world of network security, '''nothing is perfect!''' This isn’t about being perfect. It’s about controlling what we can control, and minimizing risk &amp; attack surface every chance we can. A UFC fighter makes a better bodyguard than a mall cop, regardless of the fact that they’re equally useless against a bomb or a comet.

This guide walks you through the process of setting up OpenVPN on '''pfSense'''. OpenVPN allows you to access your home network as if you were there.

All of the services we want to use require having access to this network we are placing our server on, from anywhere. This setup will make sure that all traffic from the phone is routed through the VPN with no DNS leaks, which will be important for our adblocking-via-router section that comes after.

<span id="setting-up-openvpn-within-pfsense-for-secure-access"></span>

== Setting up OpenVPN within pfSense for secure access ==

<span id="step-1-install-openvpn-client-export-package-in-pfsense"></span>
=== Step 1: Install OpenVPN Client Export package in pfSense ===

This will make it way easier for us to create the files necessary for clients to connect. You click a button and it’ll generate a file that you put on your phone or laptop. You open the OpenVPN client, import the file, put in your username &amp; password, &amp; boom – you’re set.

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:lu55028jxb9s_tmp_63b19b99.png
File:lu55028jxb9s_tmp_bb0dc07e.png
File:lu55028jxb9s_tmp_d66223f0.png
File:lu55028jxb9s_tmp_7a9f0c8.png
File:lu55028jxb9s_tmp_474565b.png
</gallery>

<span id="log-into-pfsense-1"></span>
==== 1.1 Log into pfSense: ====

* Open your browser and go to your '''pfSense''' IP address (e.g., <code>https://192.168.5.1</code> or <code>https://pfSense.home.arpa</code>).
* Log in with your credentials (default: <code>admin</code> / '''pfSense''' unless changed).

'''1.2 Install the package'''

* Go to '''System &gt; Package Manager &gt; Available Packages'''.
* Search for “openvpn-client-export”.
* Install the '''OpenVPN Client Export Utility'''.

<span id="step-2-set-up-certificates"></span>
=== Step 2: Set up Certificates ===

<span id="make-a-certificate-authority"></span>
==== 2.1 – Make a Certificate Authority ====

The Certificate Authority (CA) is what signs and verifies the server and client certificates used to establish secure connections. You don’t have to have any idea what that means to use a VPN. Here’s how we create the CA in '''pfSense''':

<div class="figure">

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:lu55028jxb9s_tmp_12971ff0.png
</gallery>

</div>
<div class="figure">

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:lu55028jxb9s_tmp_20129d0a.png
</gallery>

</div>
<div class="figure">

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:lu55028jxb9s_tmp_d4a33a40.png
</gallery>

</div>
# '''Log into pfSense:'''
# Open your browser and go to your '''pfSense''' IP address (e.g., <code>https://192.168.5.1</code> or <code>https://pfSense.home.arpa</code>).
# Log in with your credentials (default: <code>admin</code> / '''pfSense''' unless changed).
# '''Navigate to the Certificate Manager:'''
# Go to '''System &gt; Cert Manager''' in the top navigation menu.
# '''Create a New CA:'''
# Under the CAs tab, click the '''+ Add''' button to create a new Certificate Authority.
# '''Fill in the CA Details:'''
#* '''Descriptive Name:''' OpenVPN-CA (or any name you choose)
#* '''Method:''' Create an Internal Certificate Authority
#* '''Key Length:''' 4096 bits (recommended for strong security)
#* '''Digest Algorithm:''' SHA-512 (for secure hashing)
#* '''Lifetime (days):''' 3650 (about 10 years)
#* '''Distinguished Name:'''
#** '''Country Code:''' Your country’s two-letter code (e.g., US for the United States)
#** '''State or Province:''' Your state or province
#** '''City:''' Your city or locality
#** '''Organization:''' Your organization name
# '''Common Name:''' OpenVPN-CA (or another descriptive name)
# '''Save the CA:'''

<span id="creating-the-openvpn-server-certificate"></span>
==== 2.2 - Creating the OpenVPN Server Certificate ====

Next, create the server certificate that the OpenVPN server will use for secure client connections.

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:lu55028jxb9s_tmp_bfc83cc1.png
File:lu55028jxb9s_tmp_fe565fd6.png
File:lu55028jxb9s_tmp_ffd201ba.png
</gallery>

<ol style="list-style-type: decimal;">
<li><p>Navigate to the <code>Certificates</code> tab in Cert Manager.</p>
<ul>
<li>'''Add a New Server Certificate:'''</li></ul>
</li>
<li><p>Click '''+ Add/Sign''' to create a new certificate.</p></li>
<li><p>'''Fill in the Server Certificate Details:'''</p>
<ul>
<li><p>'''Method:''' Create an Internal Certificate</p></li>
<li><p>'''Descriptive Name:''' OpenVPN-ServerCert – name it something that makes it easy to identify as a '''SERVER''' certificate later for OpenVPN</p></li>
<li><p>'''Certificate Authority:''' Select OpenVPN-CA (the CA you just created)</p></li>
<li><p>'''Key Length:''' 4096 bits</p></li>
<li><p>'''Digest Algorithm:''' SHA-512</p></li>
<li><p>'''Certificate Type:''' Server Certificate.</p>
<blockquote><p>'''WARNING:''' Make sure you do not leave this set to user certificate, which is the default option.</p></blockquote></li>
<li><p>'''Lifetime (days):''' 3650</p></li>
<li><p>'''Distinguished Name:''' Match the details you used for the CA</p></li>
<li><p>'''Common Name:''' louis.chickenkiller.com (you can use whatever you put for your dynamic DNS domain name here)</p></li></ul>
</li>
<li><p>Click '''Save'''. You should now see OpenVPN-ServerCert listed under the Certificates tab.</p></li></ol>

<span id="create-a-vpn-group-for-your-vpn-users"></span>
==== 2.3 Create a VPN Group for your VPN users ====

To connect your Android phone to the VPN, create a user account with an associated client certificate.

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:lu55028jxb9s_tmp_25f3872b.png
File:lu55028jxb9s_tmp_838eb68a.png
File:lu55028jxb9s_tmp_1e4e6e6e.png
File:lu55028jxb9s_tmp_8a57a6ce.png
</gallery>

'''Log into pfSense:'''

* Open your browser and navigate to your '''pfSense''' IP address (e.g., <code>https://192.168.5.1</code> or <code>https://pfSense.home.arpa</code> or <code>pfSense.home.arpa</code>).
* Log in using your admin credentials.

'''Open User Manager:'''

Go to '''System &gt; User Manager'''.

'''Add a New Group:'''

* In the '''Groups''' tab of User Manager, click the '''+ Add''' button to create a new Group.
* '''Fill Out the Group Information:'''
** '''Group name:''' Choose a group name that makes sense for VPN users (e.g., <code>vpnusers</code>).
** Click '''Save'''.

<span id="create-a-vpn-user"></span>
==== 2.4 Create a VPN user ====

<div class="figure">

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:lu55028jxb9s_tmp_260205ab.png
</gallery>

</div>
<div class="figure">

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:lu55028jxb9s_tmp_bc14297e.png
</gallery>

</div>
<div class="figure">

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:lu55028jxb9s_tmp_cd15d249.png
</gallery>

</div>
# In the '''Users''' tab of User Manager, click the '''+ Add''' button to create a new user.
# '''Fill Out the User Information:'''
## '''Username:''' Choose a username (e.g., <code>vpnuser1</code>).
## '''Password:''' Enter a strong password.
# Add the user to the <code>vpnusers</code> group you just made.
# For '''Certificate''', check '''“Click to create a user certificate”'''. '''DO NOT FORGET TO CREATE A USER CERTIFICATE FOR THE USER.'''
# Create a name for the user certificate, such as <code>vpnuser_client_cert</code> so you can recognize it as the USER cert later.

'''BEFORE YOU HIT SAVE:'''

''Before you hit save on adding a new user account:''

# Scroll to the '''Certificates''' section of the user creation form:
# Click '''+ Add''' to generate a new certificate for this user.
# '''Configure the User Certificate:'''
## '''Certificate Authority:''' <code>OpenVPN-CA</code>
## '''Key Length:''' 4096 bits
## '''Digest Algorithm:''' <code>SHA-512</code>
# '''Save the user with the certificate:'''
# Click '''Save'''.
# Verify User Creation. You should now see the user listed under '''System &gt; User Manager &gt; Users'''.

<span id="step-3-configure-openvpn-server"></span>
=== Step 3: Configure OpenVPN Server ===

<div class="figure">

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:lu55028jxb9s_tmp_ebaec37d.png
</gallery>

</div>
<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:lu55028jxb9s_tmp_91ffd839.png
File:lu55028jxb9s_tmp_eb0c6c58.png
File:lu55028jxb9s_tmp_f489358.png
File:lu55028jxb9s_tmp_bb26515e.png
File:lu55028jxb9s_tmp_8d38ee89.png
File:lu55028jxb9s_tmp_b8064fd8.png
File:lu55028jxb9s_tmp_a7ea3e22.png
File:lu55028jxb9s_tmp_72e7ba8d.png
File:lu55028jxb9s_tmp_6ed741ea.png
</gallery>

<span id="open-the-openvpn-wizard-and-set-settings-according-to-what-you-see-below-in-section-3.2-and-in-images-above"></span>
==== 3.1 Open the OpenVPN Wizard, and set settings according to what you see below in section 3.2 and in images above ====

# '''Log into '''pfSense''':'''
# Go to '''VPN &gt; OpenVPN'''.
# Click on the '''Wizards''' tab.
# Fill out details according to what you see above. Keep in mind that when you are DONE, you will have to go back in &amp; edit settings for that VPN server that were NOT EDITABLE while you were creating the VPN.

<span id="openvpn-server-configuration"></span>
==== 3.2 OpenVPN Server Configuration ====

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:lu55028jxb9s_tmp_1be84c25.png
File:lu55028jxb9s_tmp_1ed49b8.png
File:lu55028jxb9s_tmp_a3fa0f3a.png
</gallery>

After you have finished, go back and edit that server you just made to make sure all of this matches:

# '''Description:''' openvpn server itself
#* This is for your reference only. You can name it something descriptive like “HomeVPN” or “MyVPNServer.”
# '''Protocol:''' UDP on IPv4 only
#* UDP is faster and more efficient for VPN traffic, and IPv4 only is typically sufficient unless you have a specific need for IPv6.
# '''Interface:''' WAN
#* This setting makes sure that your OpenVPN server will listen for incoming VPN connections on the WAN interface.
# '''Local Port:''' 1195
#* Default is 1194 and TOTALLY FINE. I chose 1195 because I already use 1194 for another system.
# '''TLS Authentication:''' Enabled

'''Cryptographic Settings'''

# '''DH Parameters Length:''' 4096 bits
#* Stronger than the default 2048-bit encryption.
# '''Data Encryption Algorithms:'''
#* The following algorithms are listed in the priority you selected:
#** AES-256-GCM
#** AES-128-GCM
#** CHACHA20-POLY1305
# '''Fallback Data Encryption Algorithm:''' AES-256-CBC
#* Used for compatibility if a client doesn’t support GCM encryption algorithms.
# '''Auth Digest Algorithm:''' SHA-512
#* SHA-512 provides a high level of integrity protection for your VPN packets, making sure that the data hasn’t been altered.
# '''Hardware Crypto:''' Intel RDRAND engine - RAND

'''Tunnel Settings'''

# '''IPv4 Tunnel Network:''' <code>192.168.6.0/24</code>
#* This is the virtual network that your VPN clients will use.
# '''Redirect IPv4 Gateway:''' Checked
#* This forces all client traffic through the VPN tunnel.
#* IF, for some reason, you have changed the Outgoing NAT to Manual, you'll have to add the outgoing NAT rule yourself.
# '''IPv4 Local Network:''' <code>192.168.5.0/24</code>
#* This allows VPN clients to access your local network.
# '''Allow Compression:''' Refuse any non-stub compression (Most Secure)
# '''Type-of-Service:''' Unchecked
# '''Inter-Client Communication:''' Unchecked
# '''Duplicate Connections:''' Unchecked

'''Client Settings'''

# '''Topology:''' Subnet
# '''DNS Default Domain:''' newvpn
# '''DNS Server 1:''' <code>192.168.5.1</code>
# '''DNS Server 2:''' <code>94.140.14.14</code> (AdGuard DNS)
# '''DNS Server 3:''' <code>94.140.15.15</code> (another AdGuard DNS server)

'''Advanced Client Settings'''

<ol style="list-style-type: decimal;">
<li><p>'''Dynamic IP:''' Checked</p></li>
<li><p>'''Advanced Configuration:'''</p>
<ul>
<li><p>Custom Options:</p>
<pre>tun-mtu 1200; mssfix 1160; push &quot;dhcp-option DNS 192.168.5.1&quot;;</pre></li></ul>
</li>
<li><p>'''Gateway Creation:''' IPv4 only</p>
<blockquote><p>'''For the <code>Gateway creation</code> OpenVPN server setting:''' CHOOSE ''IPv4 only'' This will save you lots of hassle and misery later! Explanation at the end of the OpenVPN section.</p></blockquote></li></ol>

<blockquote>'''NOTE:''' Let’s talk about RDRAND. This is the hardware random number generator (RNG) built into Intel processors. It’s fast, easy to use &amp; pfSense might already be using it. '''WARNING: For 99% of the people reading, this will be a total waste of time.'''
</blockquote>
<blockquote>'''1. What is RDRAND? Why care?'''
RDRAND makes random numbers using your CPU. but it’s a closed source black box. You can’t see how it works, and there have been [concerns that some random number generators might not be as random as you’d like. There are all sorts of [https://www.theregister.com/2013/09/10/torvalds_on_rrrand_nsa_gchq/ pissing matches] going on over this stuff on the internet by people way smarter than you or I.

'''Point being, if you care about privacy or you’re handling sensitive data, you might not want to rely ''solely'' on a system you can’t inspect.''' At the same time, if you’re reading this guide, you’re enough of a newbie that rdrand is not going to be how someone “gets” you.
</blockquote>
<blockquote>'''2. Why not use just RDRAND?'''
While it is fast, if the hardware random number generator fails or is compromised, your security goes down without noticing. A VPN depends on top-notch [https://www.youtube.com/watch?v=9mxE9sEGNmA randomness] for encryption, so you need more than one source of entropy to stay safe.
</blockquote>
<blockquote>'''3. How do I make it safer?'''
pfSense already mixes entropy from several sources which includes rdrand. In most cases, you’re good to go.
</blockquote>
<blockquote>'''4. Should I disable it?'''
Probably not. RDRAND is fine. Think of it as an ingredient rather than the entire thing.

'''5. THen why did you mention it?'''

The ''“uhm, akshually”'' people. They’re in the bushes, always waiting.
</blockquote>
<blockquote>'''TL;DR:''' RDRAND isn’t bad, but don’t trust it alone. Let pfSense do its thing and mix it with other entropy sources. If you’re running anything highly sensitive and don’t like trusting Intel, you can disable it—but for most people, you’ll be fine with the default settings.
</blockquote>
<span id="step-4-get-.ovpn-file-to-connect-your-phone-to-the-vpn"></span>
=== Step 4: Get .ovpn file to connect your phone to the VPN ===

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:lu55028jxb9s_tmp_5ee24475.png
File:lu55028jxb9s_tmp_cc7231bd.png
</gallery>

<span id="export-the-openvpn-client-configuration-for-your-android-device"></span>
==== 4.1 Export the OpenVPN Client Configuration for Your Android Device ====

# Go to '''VPN &gt; OpenVPN &gt; Client Export'''.
# For “remote access server,” choose the OpenVPN server you made.
# For “Host Name,” enter the URL you made on FreeDNS for dynamic DNS. In our case, this was <code>louishomeserver.chickenkiller.com</code>.
# Under '''Export Type''', choose '''Android - OpenVPN Connect'''.
# Download the configuration file (e.g., <code>vpnuser1-android.ovpn</code>).

<div class="figure">

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:image-20241106121731554.png
</gallery>

</div>
<div class="figure">

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:image-20241106121809040.png
</gallery>

</div>
<div class="figure">

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:image-20241106121822988.png
</gallery>

</div>
<div class="figure">

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:image-20241106121830675.png
</gallery>

</div>
<div class="figure">

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:image-20241106121845493.png
</gallery>

</div>
<div class="figure">

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:image-20241106121937304.png
</gallery>

</div>
<div class="figure">

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:image-20241106121943336.png
</gallery>

</div>
<div class="figure">

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:image-20241106121948680.png
</gallery>

</div>
<div class="figure">

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:image-20241106121953605.png
</gallery>

</div>
<div class="figure">

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:image-20241106122002345.png
</gallery>

</div>
<div class="figure">

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:image-20241106122017732.png
</gallery>

</div>
<div class="figure">

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:image-20241106122022741.png
</gallery>

</div>
<div class="figure">

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:image-20241106122027679.png
</gallery>

</div>
<div class="figure">

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:image-20241106122036058.png
</gallery>

</div>
<span id="import-the-configuration-into-openvpn-connect-on-android-securely"></span>
==== 4.2 Import the Configuration into OpenVPN Connect on Android – SECURELY!!!!! ====

# Transfer the <code>.ovpn</code> file to your Android device. ''DO THIS SECURELY.''
# Install the OpenVPN Connect app from the Play Store.
# Import the configuration file and connect to the VPN.

VPN connectivity can be done with a certificate alone, without a username or password. This means that ''if you misconfigured something, and this file gets into the wrong hands, '''any Tom, Dick or Harry has access to your home network!'''''

'''Don’t upload the file to public file transfer sites'''

Don’t do this. Do not store the key to your front door on megaupload.

'''Instead, do this:'''

* Connect your phone directly to your computer with a USB cable to transfer the file; simple and secure.
* Or, use an encrypted messenger you trust. Just make sure it’s actually secure, not just convenient.

'''Why the extra caution?'''

* This <code>.ovpn</code> file is sensitive. It’s part of what allows access to your server.
* If someone gets this file &amp; figures out your password, they’re in. Not good.
* And if there’s a config mistake (it happens), they might not even need the password.
* Without this file, even if someone knows your username &amp; password, they’re not getting in.

'''Treat this file like your bank details. Don’t put it on a post-it note to the 4:3 monitor in front of your Windows XP Service Pack 1 computer.'''

Don’t leave it lying around in your downloads folder.
Don’t share it casually.

The chances of someone intercepting this file and using it maliciously are low, but we don’t take unnecessary risks with security. It’s not paranoia, it’s good practice.

Do it right, and you’ll save yourself potential headaches down the road. Plus, you’ll have the satisfaction of knowing you’ve set things up properly.

<span id="edit-settings-on-openvpn-android-application"></span>

==== 4.3 Edit Settings on OpenVPN Android Application ====

# Open the OpenVPN Connect application.
# Go to the three lines in the upper left corner and tap '''Settings'''.
# Scroll down to '''Advanced Settings'''.
# Switch security level from “legacy” to “preferred”.
# Uncheck '''“DNS fallback”'''.

<blockquote>'''NOTE:''' Disabling “DNS fallback” keeps the VPN connection from going back to using non-ad-blocking(and usually google) DNS when something fails. When your setup breaks, I want you to KNOW - by way of it not working. I don’t want it to training-wheels you back to a working setup using Google’s DNS.
</blockquote>
You now have an OpenVPN server on '''pfSense''' you can connect to from anywhere in the world; your Android device will have all its traffic routed through the VPN. You’ll fully benefit from '''pfBlockerNG'''’s ad-blocking via IP blocking and DNS domain name blocking when you’re logged in through the VPN, and you’ll have access to all of the services we will be setting up for calendar, contacts, email, backup, office, home automation &amp; surveillance, business phone, password management &amp; more.

<span id="ipv4-vs-ipv4ipv6-vpn-nightmares"></span>
== IPv4 vs IPv4+IPv6 &amp; VPN nightmares: ==

Choosing IPv4 + IPv6 can cause issues. I’ve seen this cause random disconnects after about 10 minutes of connection that is miserable to figure out.

In my case, I am combining two of the worst things in the world: American residential cable broadband &amp; T-Mobile on a [https://9to5google.com/2021/11/17/pixel-6-modem-analysis/ Pixel phone]. I lose 5G when I walk under a tree, and my internet goes down more often than yours.

<span id="why-using-ipv4-ipv6-with-openvpn-for-this-setup-is-discouraged."></span>

== Why using IPv4 &amp; IPv6 with OpenVPN for this setup is discouraged. ==

Enabling both IPv4 and IPv6 may be the way to go for enterprise class connections. If you’re reading this, you might be stuck on horrible residential broadband &amp; unable to pick a better ISP. In these environments, the 1% benefit IPv6 enables

# '''NAT64/DNS64 Compatibility Issues''': Mobile networks often use NAT64/DNS64 for IPv6-only networks. This can clash with your VPN’s IPv6 routing, causing random failures.
# '''Path MTU Discovery (PMTUD) Quirks''': IPv6 relies heavily on PMTUD. If there are issues along the path, you can have connectivity problems that are hard to diagnose.
# '''ISP IPv6 Implementation''': Some ISPs (spectrum) can have less-than-great IPv6 implementations. This can lead to unstable connections when you’re trying to use both IPv4 and IPv6.
# '''Dual-Stack Timeout Issues''': When both protocols are available, your devices might try connections on both. If IPv6 is unstable, you’ll experience timeouts and apparent connection failures. '''THIS MAKES UP FOR ANY &amp; ALL POTENTIAL BENEFITS OF IPv6, WHICH YOU WILL NEVER NOTICE IN EVERYDAY USAGE.'''
# '''Carrier-Grade NAT (CGN) Interactions''': The interplay between CGN for IPv4 and IPv6 routing through your VPN can lead to connection state inconsistencies.

<span id="the-practical-solution"></span>
== The Practical Solution ==

You have two main options:

# '''Live In a Nightmare''': Dive deep into network engineering, potentially spend $150,000 backhauling fiber to your house to get around your [https://www.youtube.com/watch?v=vbHqUNl8YFk&t=37s horrible cable company].
# '''A Practical Approach''': Click “IPv4 only” in OpenVPN server settings.

Option #1 can gargle my balls.

<span id="setting-up-pfblockerng-for-ad-blocking-in-pfsense"></span>
= Setting Up '''pfBlockerNG''' for Ad-Blocking in pfSense =

<span id="why-adblock-at-the-router"></span>
=== Why adblock at the router? ===

'''''Why not'''''?? Isn’t this '''beautiful'''?

<pre>louis@happycloud:~/Downloads/frigate$ ping googleadservices.com
ping: googleadservices.com: Name or service not known</pre>
Seeing ''Name or service not known'' trying to contact a google ad server warms my heart. :D

Ad-blocking at the router level offers several advantages:

# '''Simplicity:''' Instead of installing ad-blockers on every device, you can block ads network-wide.
# '''Complete coverage:''' Blocks ads on devices where traditional ad-blockers can’t be installed (smart TVs, Android/iOS apps). Somewhere, there is probably some piece of garbage application that has an ad in it that you can’t install [https://ublockorigin.com/ ublock origin] onto. What if it were blocked from connecting at the router level?
# '''Control:''' You can manage internet connectivity and ad-blocking for all connected devices from a single point.

We’ll use two methods for blocking:

* '''IP address blocking''' - blocking <code>103.31.6.184</code>
* '''Domain name blocking''' - blocking <code>googleadservices.com</code>

This dual approach makes sure more effective ad-blocking, as it covers both static IP addresses and changing domain names associated with ad servers.

<span id="step-1-measure-our-baseline"></span>
=== Step 1: Measure our Baseline ===

<span id="install-stock-google-chrome"></span>
==== 1.1 Install [https://www.google.com/chrome/ stock Google Chrome] ====

<div class="figure">

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:lu55028jxb9s_tmp_bceadf90.png
</gallery>

</div>
No ad-blocking extensions, no privacy protections. We want to test our ROUTER’S ability to block ads – not our browser’s. The browser is going to be the ''“constant”'' here. In an ideal setup, we want to block ads at the router level (which we CAN control) in order to not see ads in random Android apps &amp; unreliable smart TVs (which we can’t always control).

You won’t always be able to block ads with certain hardware or software. And even if you can, can your boyfriend, your mother-in-law, your kids? Imagine having kids that grow up in a household with no ads. :)

Don’t use your normal web browser with all the ad-blocking stuff built-in because then we can’t tell if what we did actually worked. We’re starting by installing stock, vanilla Google Chrome, no extensions installed, and running a couple of quick tests. Something tells me Google’s business model isn’t going to provide us an ad-free web browsing experience by default…

<span id="run-adblock-dns-tests"></span>
==== 1.2 Run adblock &amp; DNS tests ====

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:lu55028jxb9s_tmp_16d96f79.png
File:lu55028jxb9s_tmp_ccf2f730.png
File:lu55028jxb9s_tmp_29860614.png
</gallery>

* [https://adblock-tester.com/ adblock-tester.com]
* [https://d3ward.github.io/toolz/adblock.html d3ward.github.io/toolz/adblock.html]  -> This project is no longer maintained and has been archived.
* [https://dnsleaktest.com/ dnsleaktest.com]

'''My Initial results:'''

* Ad-block tester: 38 points out of 100
* D3Ward Ad Block testing: 6 blocked out of 135
* DNS: Using home device ('''pfSense''' DNS resolver)

''Your mileage will vary.''

<span id="step-2-install-pfblockerng"></span>

=== Step 2: Install '''pfBlockerNG''' ===

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:lu55028jxb9s_tmp_36101b0c.png
File:lu55028jxb9s_tmp_df926adb.png
File:lu55028jxb9s_tmp_8c37d4ee.png
File:lu55028jxb9s_tmp_6863d4b9.png
</gallery>

# Log in to your '''pfSense''' web interface.
# Navigate to '''System &gt; Package Manager &gt; Available Packages'''.
# In the search bar, type '''“pfBlockerNG”'''.
# Find <code>pfBlockerNG-devel</code> and click the '''Install''' button (you want the devel version because it receives more updates &amp;, as AvE would say, is more betterer).
# Wait for the installation to complete.

<span id="step-3-configure-pfblockerng-general-settings"></span>
=== Step 3: Configure '''pfBlockerNG''' General Settings ===

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:lu55028jxb9s_tmp_e3a57841.png
File:lu55028jxb9s_tmp_bef0d3ca.png
File:lu55028jxb9s_tmp_f6e73464.png
File:lu55028jxb9s_tmp_2c627cd5.png
</gallery>

# After installation, go to '''Firewall &gt; pfBlockerNG'''.
# Under '''General Settings:'''
## Enable pfBlockerNG: Make sure this is checked.
# Click '''IP''' next to general.
# For '''Outbound Firewall Rules''', make sure both '''LAN''' and '''OpenVPN''' interfaces are selected for REJECTING.
# '''I had you set up OpenVPN''' '''''before''''' '''pfBlockerNG explicitly''' '''''because''''' '''it makes this option automatically be checked for you, but double check just in case!'''
# Click '''Save''' at the bottom.

<span id="step-4-set-up-dnsbl-dns-blacklists"></span>
=== Step 4: Set Up DNSBL (DNS Blacklists) ===

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:lu55028jxb9s_tmp_d758afc8.png
</gallery>

# Navigate to '''Firewall &gt; pfBlockerNG &gt; DNSBL'''.
# Enable DNSBL: Check this box to enable DNS-based blocking.
# DNSBL Mode: Set to '''Unbound Mode''' to use pfSense’s DNS Resolver for DNSBL.
# Go down to '''DNSBL Configuration''', make sure some random bs IP is in ''virtual IP address (LIKE 10.10.10.1), this is where we are directing requests to ad-ridden domain names to.''

<span id="step-5-add-dnsbl-feeds-ip-blocklist-feeds-lists-of-ad-domains"></span>
=== Step 5: Add DNSBL Feeds &amp; IP blocklist feeds (Lists of Ad Domains) ===

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:lu55028jxb9s_tmp_ab93a985.png
File:lu55028jxb9s_tmp_e7f85dc0.png
File:lu55028jxb9s_tmp_d117a3f0.png
</gallery>

Let me explain how these feeds work in pfBlockerNG because the interface can be intimidating for a newbie.

The feeds tab has two main sections: IP address feeds at the top (for blocking specific IPs) and DNS feeds at the bottom (for blocking domain names like googleadservices.com).

When you’re looking at the feeds, you’ll see these checkboxes and plus signs that can be a bit confusing. Here’s what they mean:

* If you see a checkbox on the left, that means it’s a GROUP of feeds. If you see a blue checkbox next to “PRI1” that means all the feeds under that group are already enabled.
* Individual feeds will have their own checkboxes to show if they’re active.
* The plus signs let you add new feeds to your configuration.

When you want to add feeds, click the plus sign to add the feed.

For IP blocklists, make sure the action is set to '''“Deny Both”'''.

For DNS blocklists, set the action to '''“Unbound”'''.

Even if you see something’s already checked, sometimes clicking “Enable All” can catch feeds that weren’t properly activated. I’ve had weird situations where I thought I added everything in a group but missed some - the interface isn’t always super clear about what’s actually enabled.

For what to block: I avoid blocking things like Tor or torrent trackers. Why would you block that? That’s like DDoSing Pornhub - they’re giving you free stuff! One of them blocks AWS, avoid that unless you want non-functional internet (sadly the world runs on AWS whether we like it or not).

<blockquote>It is very easy to block too much and then not be able to log into youtube, receive email, visit your bank, etc. More isn’t better here.
</blockquote>
# Go to '''Firewall &gt; pfBlockerNG &gt; Feeds'''.
# Scroll to the '''DNSBL Feeds''' section.
# Add multiple feeds by clicking on different categories and enabling relevant lists.
# For each selected feed:
#* For DNS block lists, set “Action” to '''Unbound'''.
#* For IP lists, set “Action” to '''Deny Both'''.
# There is a blue “ENABLE ALL” method at the bottom that will often save you a lot of time.
# Recommended categories to add:
#* Easylist
#* Malicious
#* Phishing
#* Malware
#* Suspicious
#* Trackers
#* Spam (for email)
# '''Avoid adding feeds that might block legitimate services (e.g., AWS, public DNS servers, Tor).'''
# After selecting feeds, click '''Save''' to apply these DNSBL lists.
# Don’t enable/turn them on one by one. When you click on a list of feeds, note the blue '''“enable all”''' button. ''Don’t be like Louis of 2018 &amp; toggle each line to “on” manually like an idiot (I actually did this :’( )''

<span id="step-6-update-and-apply-lists"></span>
=== Step 6: Update and Apply Lists ===

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:lu55028jxb9s_tmp_88427bbb.png
</gallery>

# Navigate to '''Firewall &gt; pfBlockerNG &gt; Update'''.
# Select '''“Force”''' option.
# Set '''“Reload”''' option to '''“All.”'''
# Click '''“Run”''' to download and update all lists (both DNSBL and IP lists).

''This process can take a while.''

<span id="step-7-testing-and-verifying-ad-blocking-effectiveness"></span>
=== Step 7: Testing and Verifying Ad-Blocking Effectiveness ===

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:lu55028jxb9s_tmp_cd49ce7b.png
File:lu55028jxb9s_tmp_82193a4c.png
</gallery>

# Clear cache and cookies in your test browser.
# Revisit the ad-blocking test sites:
## [https://adblock-tester.com/ adblock-tester.com]
## [http://d3ward.github.io/toolz/adblock.html d3ward.github.io/toolz/adblock.html] ->  This project is no longer maintained and has been archived.

'''Expected results:'''

* Ad-block tester: Improved score (e.g., 78 out of 100)
* D3Ward Ad Block testing: Many more blocked (e.g., 119 out of 135)

<span id="step-9-implement-adguard-dns"></span>

=== Step 9: Implement AdGuard DNS ===

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:lu55028jxb9s_tmp_ae2b0b8d.png
File:lu55028jxb9s_tmp_1ca05f7d.png
File:lu55028jxb9s_tmp_b129b41f.png
File:lu55028jxb9s_tmp_18e54ec0.png
</gallery>

# Visit [https://adguard-dns.io/en/public-dns.html adguard-dns.io] and go to the '''“Routers”''' section.
# Copy the DNS server addresses that block ads and trackers.
# In '''pfSense''', go to '''System &gt; General Setup'''.
# Uncheck '''“Allow DNS server list to be overridden by DHCP/PPP on WAN.”'''
# Remove existing DNS servers and add the AdGuard DNS servers. '''Use what is on AdGuard’s site: at the time of this writing, they were as follows. Only use the below servers if you see them on''' [https://adguard-dns.io/en/public-dns.html '''adguard-dns.io''']:
## Primary DNS: <code>94.140.14.14</code>
## Secondary DNS: <code>94.140.15.15</code>
# You checked [https://adguard-dns.io/en/public-dns.html AdGuard’s site] rather than copy &amp; paste from here, right? RIGHT?
# Save changes.

<span id="step-10-configure-the-dns-resolver"></span>

=== Step 10: Configure the DNS Resolver ===

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:lu55028jxb9s_tmp_18e54ec0.png
File:lu55028jxb9s_tmp_ac1bdd90.png
File:lu55028jxb9s_tmp_a68efb7e.png
</gallery>

# Go to '''Services &gt; DNS Resolver'''.
# Enable DNS Resolver: make sure this is checked.
# Click '''Enable Forwarding Mode'''.
# Save and apply changes.
# Reload the DNS Resolver service.

<span id="step-11-verify-adblocking-from-desktop"></span>
=== Step 11: Verify adblocking from Desktop ===

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:lu55028jxb9s_tmp_253defc2.png
File:lu55028jxb9s_tmp_6f775d7c.png
File:lu55028jxb9s_tmp_7763a7c0.png
File:lu55028jxb9s_tmp_51675dc6.png
File:lu55028jxb9s_tmp_2de8777b.png
</gallery>

# Clear DNS cache and browser data.
# Rerun the ad-blocking tests.
# Visit [https://dnsleaktest.com/ dnsleaktest.com] and run an extended test to confirm you’re using AdGuard DNS. You should see something like the figure above. Your DNS should be DIFFERENT than it was before! If not, something went wrong.
# Redo your adblock test:
#* [https://adblock-tester.com/ adblock-tester.com]
#* [https://d3ward.github.io/toolz/adblock.html d3ward.github.io/toolz/adblock.html]
# You should see adblocking become even more better, or more betterer as [https://www.youtube.com/@arduinoversusevil2025 AvE] would say, than what you had prior to installing pfBlockerNG, depending on the feeds you’ve chosen.

<span id="step-13-verify-adblock-on-mobile-via-vpn"></span>
=== Step 13: Verify adblock on mobile via VPN ===

<div class="figure">

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:image-20241106123802153.png
</gallery>

</div>
<div class="figure">

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:image-20241106123837196.png
</gallery>

</div>
<div class="figure">

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:image-20241106124123482.png
</gallery>

</div>
<div class="figure">

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:image-20241106124131570.png
</gallery>

</div>
<div class="figure">

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:image-20241106124141942.png
</gallery>

</div>
<div class="figure">

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:image-20241106124249587.png
</gallery>

</div>
<div class="figure">

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:image-20241106124207433.png
</gallery>

</div>
To make sure ad-blocking works on mobile devices connected through VPN:

# Clear browser data on your phone.
# '''Disconnect from the VPN we attached to earlier.'''
# Visit the following websites and note the results:
#* [https://adblock-tester.com/ adblock-tester.com] – should have horrible results
#* [https://d3ward.github.io/toolz/adblock.html d3ward.github.io/toolz/adblock.html] – also horrible results
#* [https://dnsleaktest.com/ dnsleaktest.com] – should show AdGuard DNS, same as what you saw in the above figure on your PC
# Go over to the OpenVPN app &amp; connect to VPN
#* [https://adblock-tester.com/ adblock-tester.com] – should have better results
#* [https://d3ward.github.io/toolz/adblock.html d3ward.github.io/toolz/adblock.html] – should have better results
#* [https://dnsleaktest.com/ dnsleaktest.com] – should show your mobile provider’s DNS servers

Double-check that you’re using the '''pfSense''' DNS on <code>dnsleaktest.com</code> &amp; NOTHING ELSE!! You do not want your ISP’s server, or anyone else’s server, to show up. If in doubt, research the IP address &amp; hostname of the DNS that is coming up.

<ol start="5" style="list-style-type: decimal;">
<li>Compare results to those without a VPN connection.</li></ol>

'''Expected results:'''

* Much more ad-blocking on mobile when connected to VPN
* Confirmation that you’re using AdGuard DNS through the VPN

<div class="figure">

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:image-20241106125051953.png
</gallery>

</div>
<span id="step-14-verify-vpn-allows-connectivity-to-home-network."></span>
=== Step 14: Verify VPN allows connectivity to home network. ===

Try to visit your router’s IP address https://192.168.5.1/ once you have connected to the VPN – and '''make sure you are connected to the CELLULAR network only, not your home Wi-Fi!!'''

Congratulations; you’ve set up an ad-blocking system that blocks a ton of ads before your internet connection even wastes bandwidth loading them, for all devices on your network. Blocking ads in a browser using uBlock Origin is fun, but '''nothing compares to the feeling of blocking ads they think you can’t block. It’s beautiful. :D''' This means that even inside of Android apps that have ads, you can block them all—it just takes the right feed. :D

'''REMEMBER: THIS IS YOUR JOURNEY!!! FIND THE FEEDS THAT MAKE YOU HAPPY, YOU DO NOT HAVE TO USE THE SAME ONES THAT I DID!'''

<span id="installing-ubuntu-server-with-raid-1-lvm-and-luks-encryption"></span>
= Installing Ubuntu Server with RAID 1, LVM, and LUKS Encryption =

Now it’s time to install the operating system on our host server. I’ll walk you through the process of installing Ubuntu Server with a nice configuration including RAID 1 for boot drive redundancy, encrypted LVM for flexibility in expanding storage if we move this setup to a larger set of drives, and LUKS encryption for security. This setup makes sure your server can boot even if one drive fails, while keeping your data secure. Even if someone breaks into your house &amp; steals all of your stuff, all they have is encrypted crap. Unless they’re the NSA, in which case you’re screwed, but if you’re reading this guide, you’re probably not that important.

<span id="installing-ubuntu-linux"></span>
== Installing Ubuntu Linux ==

For our server’s operating system, we’re going with Ubuntu Linux. Why Ubuntu? If you’re watching this, you’re probably more of a newbie than an expert. Ubuntu is user-friendly, has good documentation, and has a huge community ready to help. It’s widely renowned as the first “newbie friendly” GNU/Linux distribution, dating back to 2006 when it was one of the few distros that ''[https://distrowatch.com/dwres.php?resource=review-ubuntu didn’t require torturing yourself with ndiswrapper to get wifi working]''. Robert Storey put it best:

“''The huge collection of Linux/BSD systems listed on DistroWatch is a testimonial to how difficult it is to make a decision. However, after spending weeks trying to get XYZ distro to recognize your wireless card, it’s really nice to have an OS that just works.”''

Imagine having a laptop as your only computer, before smartphones with tethering were widely available. You don’t have access to a wired connection. Where were you getting your drivers from? Maybe you do have access to a wireless connection, but your only CAT5 cable is 5 feet long. And your router is in an un-air-conditioned garage. In the middle of summer. So you go to your 98°F garage, sit on the floor, googling only to find a plethora of threads where elitist douchebags tell you to RTFM to get wifi to work.

And they wonder why people used closed source operating systems…

In 2005, the concept of anything in GNU/Linux ''“just working”'' was a joke. If you wanted to burn a CD you had to set up something called [https://forums.justlinux.com/showthread.php?29773-ide-scsi-emulation-isn-t-working SCSI emulation] to use the optical drive on your computer. From the ground up, GNU/Linux was fundamentally not designed for normal people. Ubuntu changed that in a radical way and continues to have a reputation for being a newbie-friendly ''“gateway drug”'' to GNU/Linux. It’s not the best and it has its flaws, but it is designed and developed with ease of use for normal people in mind. For a beginner’s guide, that matters.

<span id="why-not-arch-or-gentoo"></span>
=== Why Not Arch or Gentoo? ===

I use Arch Linux now, SuSE from 2002-2004, and Gentoo from 2004-2015. I enjoy making my life difficult for no good reason. In my 30s, I’ve come to realize that I derive sick pleasure from making my life difficult for no good reason; but I wouldn’t recommend that for beginners ''(or anyone)''. With Ubuntu, you get a system that’s easy to set up and maintain without the extra hassle, designed to be as idiot-proof as possible, and designed for normal humans to use. If you wish to use another distro, '''''GO FOR IT!''''' There is NO one ''“right way”'' to do any of what I am doing here!

<span id="installing-with-raid-1-choosing-your-os-drive"></span>
=== Installing with RAID 1: Choosing Your OS Drive ===

We are going to be using RAID 1. RAID 1 is a mirroring setup, where we use two drives for the operating system instead of one. This means one of the drives can completely fail and the server continues running. I would suggest that you find not one, but TWO SSDs for this purpose. We will be using <code>MDADM</code> for RAID. Ubuntu allows you to do this upon install without having to edit configuration files.

<span id="why-software-raid-using-mdadm-instead-of-hardware-raid-with-a-raid-controller-card"></span>
==== Why software RAID using MDADM instead of hardware RAID with a RAID controller card? ====

RAID controller cards are for people with datacenters that have hundreds of drives and need maximum performance/resilience for specific applications, that want the task of managing these drives separate from the software running the computer. This was also very useful back when machines were powered by Pentium 1 processors.

<blockquote>'''NOTE:''' Some hardware RAID controllers will give you improvements in performance, but it’s not worth the downside. There are controllers where when they fail, you have to replace it with the exact same controller for your setup to work again - aka, digiorno all over again. Using software RAID like MDADM means you can take drives out of a pentium 4 and put them into a macbook and it’ll just detect it &amp; work.
</blockquote>
It is 2024, and even a ten-year-old computer will do software RAID just fine with no perceivable penalty in performance.

<span id="why-not-use-raid-built-into-my-motherboard"></span>
==== Why not use RAID built into my motherboard? ====

That is called ''“fake RAID.”'' Fake RAID is cancer. It is not “hardware” RAID, it is just software RAID by another name.

When you create a RAID array using the garbage built into your motherboard, the RAID configuration is sometimes stored in a proprietary format that is only readable by that specific manufacturer’s RAID implementation. I used the word ''“sometimes”'' because it depends on your system. I have no idea what system you have. I want ALL of the people reading this to have a system that works if they transfer these drives to another system, not ''“some”'' of you. It costs you nothing to use <code>mdadm</code>, which offers certainty of compatibility when you transfer these drives to other hardware.

When certainty &amp; uncertainty have the same price, all other things being equal, I’ll take the certainty!

MDADM software RAID is a standardized system that transfers across computers – I am not using hardware RAID, I am not using whatever RAID is in the BIOS of your computer, because I have no idea what they are using or whether it is something standard or something that will be aggravating later. If you have to take these drives and put them in another computer, there will be less hassle using software RAID than there is using hardware RAID, it’s literally plug and play ''(well, you may have to use a liveCD to run <code>grub-install</code> to register the bootloader with the new machine’s UEFI, but… The RAID part will work at least!).''

<span id="drive-recommendation-for-os"></span>
==== Drive recommendation for OS: ====

We’re going to have two drives in RAID 1. '''''You can use more if you like – RAID 1 need not be two drives!''''' I like Micron SSDs; they have always had consistently lower failure rates than Samsung’s budget “EVO” line for me with regards to NVME devices. I’ve RMA’d the same 2 TB Samsung EVO 970 five times now… Five… Times. You can get two budget 4 TB SSDs for under $500 now – I recommend ''[https://www.crucial.com/ssd/p3/CT4000P3SSD8 these]''.

We are going to be using these SSDs for virtual machines that perform many tasks. Here are some of the storage-intensive ones:

* Self-hosted mail. Your inbox may be 50+ GB like mine.
* Complete phone backup of everything – can easily eclipse 2 terabytes. Mine is 1.4.
* FreePBX phone system – call recordings over time can go over 50 GB easily.

I suggest buying drives for your operating system disk that are considerably fast and have enough space to store all of this. With regards to security camera recordings, and the backup of your 40 terabytes of recipes stored as .mkv files – that, we’ll do on an array of hard drives. You don’t need to get SSDs.

<span id="raid-is-not-a-backup"></span>

== RAID IS NOT A BACKUP! ==

<blockquote>'''IMPORTANT NOTICE: RAID 1 IS NOT A BACKUP!'''
Many people incorrectly believe that RAID 1 is a “backup.” It is not! RAID 1 sets up your machine so that the operating system is installed on TWO drives rather than one, with each drive being an exact mirror of the other. This way, if one drive fails while you’re using your server, it will still run. Think of RAID 1 like the green goo you can put in your tire to plug up a hole, or a spare wheel, allowing you to limp to a service center for repairs.
</blockquote>
Here are a few reasons why RAID 1 is not a backup:

# Backups allow you to restore your system if you accidentally mess something up. RAID 1 is a perfect mirror, so it applies to everything you break.
# RAID 1 means you’re attaching two hard drives to your computer to install the operating system on instead of one. These drives are both connected to the same computer. If your computer’s power supply fails and sends incorrect voltages to the drives, both get fried.
# When one drive in a RAID 1 array fails, the other often fails soon after, especially if they’re the same brand and were purchased at the same time.
# RAID 1 works so well that you might not notice when one drive fails until the second one also fails, leaving you with no data.

<blockquote>'''NOTE:''' MDADM does work well enough that you won’t tell when a drive fails. Later in this guide we’re going to set it up so that your machine is constantly checking &amp; emails you the moment there is any issue with your drive using mdadm’s monitor command.
</blockquote>
<span id="step-by-step-installation-guide"></span>
== Step-by-Step Installation Guide ==

'''What you should have'''

* Two identical SSDs (e.g., Samsung 870 EVO 250GB), but bigger will be better here since we’ll be using this to backup everything on your phone + many other things.
* A USB drive to put the Ubuntu installation image on
* An old computer to use as a server (even a 10-year-old desktop or laptop can work)

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:lu55028jxc7f_tmp_c21a542a.png
File:lu55028jxc7f_tmp_133ad038.png
File:lu55028jxc7f_tmp_c5e0db26.png
File:lu55028jxc7f_tmp_1eafd48e.png
File:lu55028jxc7f_tmp_6c5ec0f7.png
File:lu55028jxc7f_tmp_51afdc6d.png
File:lu55028jxc7f_tmp_bd4536d.png
File:lu55028jxc7f_tmp_f70b5843.png
</gallery>

<span id="prepare-the-installation-disk"></span>
=== 1. Prepare the Installation Disk ===

''Warning: This process will erase everything on the USB drive.''

# Insert a USB flash drive (at least 4GB in size) into your computer.
# Go to [https://ubuntu.com/server ubuntu.com] and download the LTS (Long Term Support) version of Ubuntu Server.
# Use one of the following methods to write the Ubuntu image to the USB drive:

'''Windows:'''

# Download and install Rufus.
# Open Rufus and select your USB drive.
# Click the '''“SELECT”''' button and choose the unzipped .img file you downloaded.
# Click '''“Start”''' and let Rufus create the bootable USB.

'''GNU/Linux or macOS:'''

<ol style="list-style-type: decimal;">
<li><p>Open the terminal and type the following command:</p>
<pre>sudo fdisk -l</pre></li>
<li><p>Make note of drives in the system.</p></li>
<li><p>Plug in the flash drive.</p></li>
<li><p>Open the terminal and type the following command again:</p>
<pre>sudo fdisk -l</pre></li>
<li><p>Make note of the drive that was not present before.</p></li>
<li><p>Double-check size/brand/model to make sure this new device is the device you plugged in.</p></li>
<li><p>Run the following, replacing <code>/dev/sdX</code> with your drive, and replace the <code>ubuntu-server.iso</code> file with the filename of your image file. Make sure you use the right PATH, that is the directory your image is in.</p>
<pre>sudo dd if=/path/to/ubuntu-server.iso of=/dev/sdX bs=4M status=progress</pre></li></ol>

Your bootable USB drive with Ubuntu Server Linux is now ready for use!

<span id="boot-from-the-usb-drive"></span>
=== 2. Boot from the USB Drive ===

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:lu55028jxc7f_tmp_911d702.png
File:lu55028jxc7f_tmp_a33d9a7f.png
</gallery>

# Insert the USB drive into your server.
# Power on the server and enter the boot menu (usually by pressing '''F12''' or another function key).
# Select the '''UEFI option''' for your USB drive.

<span id="begin-the-ubuntu-server-installation"></span>
=== 3. Begin the Ubuntu Server Installation ===

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:lu55028jxc7f_tmp_5d0eeccb.png
File:lu55028jxc7f_tmp_c5b32782.png
File:lu55028jxc7f_tmp_14d81229.png
</gallery>

# Choose '''“Try or Install Ubuntu Server”''' from the boot menu.
# Select your language and keyboard layout.
# Choose '''“Install Ubuntu Server”''' (not the minimized version).
# Select '''“Search for third-party drivers”''' for better hardware support. Don’t check this box if you want to ''[https://stallman.org/stallman-computing.html live Richard Stallman’s ethics]''. Check this box if you want to reduce the chances of random things in your computer not working. I check the box. I’m going to hell, I know….

<span id="configure-network"></span>
=== 4. Configure Network ===

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:lu55028jxc7f_tmp_b1e98c52.png
</gallery>

<span id="why-a-static-ip"></span>
==== 4.1 Why a Static IP? ====

We are going to set up a server that we are going to consistently access. This means we always want it to be at the same place.

Imagine trying to deliver mail to someone who lives on 20 Main Street today, and 90 Chandler Avenue tomorrow. Imagine trying to frequent a restaurant whose address changes every week. It would be annoying, inconvenient, and perhaps downright impossible.

We want our server to always be at the same address. The “D” in “DHCP” means “dynamic” – as in, changing. We don’t want that. We want a “static” IP, meaning it does NOT change.

When setting up your server, we need to give it a static IP, so we always know where to find it, and it never changes. How do we know what IP to give it? Go back to '''pfSense'''’s DHCP server configuration page &amp; you can find it by going to '''Services —&gt; DHCP Server'''. The ''“subnet range”'' tells you the list of available IPs. Keep in mind that you cannot use the IP address of your '''pfSense''' router here.

* '''Router Gateway''': My router’s IP is 192.168.5.1. This is the gateway address.
* '''Address Pool Range''': My address pool range is from .15 to .245, leaving .246 to .254 and .2 to .14 available. This setup provides a buffer of IPs for servers and other devices.

'''Why the Buffer?''' I don’t want any conflicts where someone plugs in their computer while mine is rebooting and steals my IP. We will be setting up STATIC MAPPINGS so that nobody else can grab the IP address of my server – the IP we choose for our server will be reserved for our server’s specific network interface card and not some hated brother in law that thinks he’ll play games when your spouse has him over. However, this is still good practice.

<span id="choosing-a-static-ip"></span>
==== 4.2 Choosing a Static IP ====

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:lu55028jxc7f_tmp_44432cac.png
File:lu55028jxc7f_tmp_feed77f5.png
File:lu55028jxc7f_tmp_93f4316.png
File:lu55028jxc7f_tmp_bda461e0.png
File:lu55028jxc7f_tmp_4d5798a7.png
File:lu55028jxc7f_tmp_c59fcb59.png
</gallery>

For my servers, I pick an IP between 192.168.5.2 and 192.168.5.14. This ensures no one else can sneakily take my server’s IP while it’s rebooting.

# In your '''pfSense''' router, go to '''Services &gt; DHCP Server'''.
# Understand your subnet. For example, <code>192.168.5.0/24</code> covers IPs from <code>192.168.5.1</code> to <code>192.168.5.254</code>
# Your router’s IP is typically <code>192.168.5.1</code>. We can’t use that. Since we made the address DHCP pool range <code>192.168.5.15</code> <code>192.168.5.245</code>, this means that we have <code>192.168.5.2</code> through <code>192.168.5.14</code> free – no computer connecting with DHCP (which is the default for 99.9999% of all network devices in your home) will be using these, so they’re free for the taking.
# Choose the network interface that’s connected (usually the one that has already received an IP via DHCP).
# Change the configuration from DHCP to Manual:

* '''IP Address:''' Choose an address outside your DHCP pool (e.g., <code>192.168.5.2</code>)
* '''Subnet:''' Usually <code>255.255.255.0</code> (or /24 in CIDR notation)
* '''Gateway:''' Your router’s IP (e.g., <code>192.168.5.1</code>)
* '''Name servers:''' Use your router’s IP as the DNS server


'''''Please note: if you skip step 4 by choosing Continue without network, you not be able to set up your internet connection later.'''''

<span id="prepare-the-drives"></span>

=== 5. Prepare the Drives ===

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:lu55028jxc7f_tmp_90d50cc3.png
File:lu55028jxc7f_tmp_ae1aab84.png
File:lu55028jxc7f_tmp_6fb6a48d.png
File:lu55028jxc7f_tmp_fc776e25.png
File:lu55028jxc7f_tmp_e4bd6c3a.png
File:lu55028jxc7f_tmp_41dc80bd.png
File:lu55028jxc7f_tmp_31ba1cbc.png
</gallery>

<span id="format-the-drives"></span>
==== 5.1 Format the drives ====

# In the installer, locate your two SSDs (ignore the USB installer drive).
# For each SSD:
#* Select the drive and choose '''“Reformat”'''.
#* Select '''“Use as boot device”''' – this will create an EFI partition on each.

<span id="configure-efi-partitions"></span>
==== 5.2 Configure EFI Partitions ====

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:lu55028jxc7f_tmp_be3ce88b.png
File:lu55028jxc7f_tmp_4b9aaa78.png
File:lu55028jxc7f_tmp_5086d648.png
File:lu55028jxc7f_tmp_8d8ec28c.png
File:lu55028jxc7f_tmp_8f536396.png
File:lu55028jxc7f_tmp_ee4b3eb0.png
File:lu55028jxc7f_tmp_84fae9c1.png
</gallery>

For each SSD:

* Locate the automatically created EFI partition (usually 1GB).
* Edit the size to '''512M'''.
* Make sure it’s set to mount at <code>/boot/efi</code>.

<span id="create-boot-partitions-for-raid"></span>

==== 5.3 Create Boot Partitions for RAID ====

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:lu55028jxc7f_tmp_5080ad97.png
File:lu55028jxc7f_tmp_5f9b3efa.png
File:lu55028jxc7f_tmp_c834b835.png
File:lu55028jxc7f_tmp_d73e5ee8.png
File:lu55028jxc7f_tmp_8ace6542.png
</gallery>

# On each SSD:
#* Create a new '''1GB''' partition.
#* DO NOT FORMAT IT. CHOOSE '''"Leave unformatted"'''.
#* '''DO NOT CHOOSE A MOUNT POINT.''' This is important for setting up RAID 1 later.
# Set Up RAID 1 for <code>/boot</code>

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:lu55028jxckj_tmp_23af8abb.png
File:lu55028jxckj_tmp_f4ef2715.png
</gallery>

# Select '''“Create software RAID (md)”'''.
# Choose both 1GB partitions you just created (one from each SSD).
# Set RAID Level to '''“RAID 1 (mirrored)”'''.
# Name it '''“bootraid”''' or something meaningful to you.
# Select '''“Create”''', hit enter.

<span id="create-root-partitions-for-raid"></span>

==== 5.4 Create Root Partitions for RAID ====

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:lu55028jxckj_tmp_d09fc3d7.png
File:lu55028jxckj_tmp_972a066a.png
File:lu55028jxckj_tmp_4a9745c7.png
File:lu55028jxckj_tmp_7f935e0e.png
</gallery>

# On each SSD:
#* Create a partition using all remaining space. Don’t fill in the “size” text box; it will automatically use the rest of the space on the drive.
#* '''DO NOT FORMAT IT. CHOOSE '''"Leave unformatted"'''.'''
#* '''DO NOT CHOOSE A MOUNT POINT.''' This is important for setting up RAID 1 later.

<span id="set-up-raid-1-for-root"></span>

==== 5.5 Set Up RAID 1 for Root ====

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:lu55028jxckj_tmp_a4a5a576.png
File:lu55028jxckj_tmp_40dcb02f.png
</gallery>

# Select '''“Create software RAID (md)”''' again.
# Choose both large partitions you just created.
# Make sure RAID Level is set to '''“RAID 1 (mirrored)”'''.
# Name it '''“osdriveraid”''' or something meaningful to you.
# Go to '''“Create”''' &amp; hit enter.

<span id="configure-the-boot-partition"></span>
==== 5.6 Configure the /boot Partition ====

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:lu55028jxckj_tmp_664f6137.png
File:lu55028jxckj_tmp_10955270.png
</gallery>

# Select the '''“bootraid”''' you created.
# Format it as '''ext4'''.
# Set mount point to <code>/boot</code>.

<span id="set-up-lvm-on-root-raid"></span>
==== 5.7 Set Up LVM on Root RAID ====

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:lu55028jxckj_tmp_83c21b25.png
File:lu55028jxckj_tmp_1514c051.png
</gallery>

# Select the '''“osdriveraid”''' you created.
# Choose '''“Create volume group”'''.
# Name it '''“ubuntuvolumegroup”''' or something meaningful to you.
# When selecting the device for the LVM, you’ll encounter [https://bugs.launchpad.net/subiquity/+bug/2062102 a bug in the installer]: &gt; ''The installer will show multiple devices without clear identifiers. This is a known issue that persists in the non-beta release of a stable, mission very important server operating system. Welcome to the world of open source software; this is part of the fun of using open source! Remember: '''it wouldn’t be open source if it worked!'''''
# To select the correct device:
#* Look for the option that’s around the size of your install (e.g., 231 GB for 250 GB SSDs).
#* Choose the largest option, which should correspond to your RAID 1 array for the root partition.
#* Ignore the smaller sizes, as they likely represent other partitions or devices.
#* Pray.
# After selecting the correct device, proceed with creating the volume group.

<span id="create-encrypted-volume"></span>

==== 5.8 Create Encrypted Volume ====

# With the LVM volume group selected, choose '''“Create encrypted volume”'''.
# Set a strong password. Consider using a password manager.
# It’s recommended not to create a recovery key, as this could be a potential security risk.
# Optionally, create a recovery key. If you do this, realize '''the recovery key can be used to decrypt your volume'''. ''Don’t do this unless you have a place to hide it that not even your cat can get to!''

<span id="create-logical-volume-for-root"></span>

==== 5.9. Create Logical Volume for Root ====

# Select the encrypted volume you just created.
# Choose '''“Create logical volume”'''.
# Name it '''“ubunturootvolume”''' or something meaningful to you.
# Use the maximum available size.
# Format it as '''ext4'''.
# Set the mount point to <code>/</code> (root).

<span id="review-and-confirm"></span>
==== 5.10 Review and Confirm ====

# Double-check your configuration. For two 250 GB SSDs, it should look like this:
#* Root (<code>/</code>): ~231GB on encrypted LVM which is on RAID 1
#* <code>/boot</code>: ~1GB on RAID 1
#* <code>/boot/efi</code>: 512MB on each SSD
# If everything looks correct, click '''“Done”'''.

<span id="complete-the-installation"></span>
==== 5.11 Complete the Installation ====

# Carefully review the summary one last time. ''Remember we are erasing everything on these drives, to a point where even Rossmann Repair can’t recover it. If you create an encrypted volume, write over it, and then want the data back… good luck with that one.''
# If you’re sure you want to proceed, click '''“Continue”'''.
# Follow the remaining Ubuntu Server installation prompts.
# Set up your username.
# Install OpenSSH server.

<blockquote>'''NOTE:''' Installing OpenSSH allows you to remotely access your machine to install things, use it, mess with it, etc, rather than sit in front of your server in your unairconditioned garage when it’s 117f outside. When you see me on video installing things via terminal, I am almost never in front of the actual machine(or vm) I am using, I am remoting in using ssh.
</blockquote>
<blockquote>'''NOTE''': Do not install Docker via Snap in the next menu when it asks you to. We will install Docker later, and it won’t be the [https://www.reddit.com/r/docker/comments/shztqs/wow_docker_works_a_lot_better_when_you_dont_have/ miserable snap version of DOCKER]_. If you install Docker using Snap accidentally, this is understandable. If you install docker via snap by CHOICE, you’ll be in hell, &amp; you’ll have earned it.
</blockquote>
<span id="reboot-log-in"></span>
==== 5.12 Reboot &amp; log in ====

# Click reboot now at the end.
# Once it is done shutting down Ubuntu Linux, unplug the installation USB.
# When it boots up, it will ask for the encryption password to unlock the root partition, type this in.

<div class="figure">

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:image-20241106140227789.png
</gallery>

</div>
<div class="figure">

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:image-20241106140354651.png
</gallery>

</div>
<div class="figure">

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:image-20241106140427599.png
</gallery>

</div>
<div class="figure">

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:image-20241106140457610.png
</gallery>

</div>
<div class="figure">

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:image-20241106140554944.png
</gallery>

</div>
<span id="set-up-static-ip-mapping-in-pfsense-post-installation"></span>
==== 5.13 Set Up Static IP Mapping in pfSense (Post-Installation) ====

<span id="set-up-static-ip-mapping-in-pfsense"></span>
==== Set Up Static IP Mapping in pfSense ====

# Log into your pfSense router.
# Go to '''Diagnostics &gt; ARP Table'''.
# Find the MAC address associated with your server’s IP (e.g., <code>192.168.5.2</code>). Mine was <code>e0:d5:5e:a8:7f:b5</code>.
# Go to '''Services &gt; DHCP Server'''.
# Scroll to the bottom and click '''“Add static mapping”'''.
# Enter the MAC address and IP address of your server.
''Figure 17: This is what my setup looks like when I’m done configuring my partition structure. Yours should resemble mine. Ubuntu makes it as difficult as possible to use encrypted LVM with RAID 1 on boot devices, but we can beat their interface with some good ol’ ingenuity.''
# Give it a descriptive name (e.g., “Happy cloud server static IP”).
# Save and apply changes.

<span id="identifying-devices-on-your-network"></span>
== Identifying Devices on Your Network ==

Let’s take a quick break to discuss the importance of '''static mappings''', '''hostnames,''' and the '''DNS resolver.'''

What you type into the <code>hostname</code> field when setting the '''DHCP static mapping''' in '''DHCP server settings''' is what you can use to connect to the device instead of the IP address. For instance, if you set the hostname to <code>happycloud</code>, instead of having to type <code>192.168.5.2</code> to connect to this device, you can type <code>happycloud.home.arpa</code>.

By default, on pfSense installations, the '''default domain''' is <code>home.arpa</code>. When you combine the <code>hostname</code> of <code>happycloud</code> with the <code>domain</code> of <code>home.arpa</code>, you get <code>happycloud.home.arpa</code>.

This is more convenient for connecting to devices because it is easier to remember <code>happycloud</code> than it is to remember <code>192.168.5.2</code> for sane people, who reserve their brains for useful data rather than [https://www.youtube.com/watch?v=Z0DF-MOkotA&t=874s useless macbook trivia]. Further, similar to dynamic DNS, if you change the IP address of this server later, all of your services &amp; bookmarks that point to this server do not have to be changed!

'''You can name your servers however you want! You can choose IP addresses for your servers however you want! I will be using the same IP addresses &amp; hostnames/domains throughout this guide so it is easy to follow, but you don’t HAVE to follow mine!'''

<div class="figure">

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:image-20241106141247324.png
</gallery>

</div>
<div class="figure">

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:image-20241106141458662.png
</gallery>

</div>
<div class="figure">

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:image-20241106141624092.png
</gallery>

</div>
<div class="figure">

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:image-20241106141726132.png
</gallery>

</div>
<span id="why-isc-dhcp-matters-in-pfsense-and-how-to-set-it-up"></span>
= Why ISC DHCP Matters in pfSense (and How to Set It Up) =

The world wants you to switch to [https://www.netgate.com/blog/netgate-adds-kea-dhcp-to-pfsense-plus-software-version-23.09-1 Kea DHCP], but there’s a very good reason we’re using ISC instead. It does something important that new DHCP server [https://forum.netgate.com/topic/184398/kea-dhcp-missing-register-dhcp-leases-in-dns-resolver/7 doesn’t]. Let’s get into it.

<span id="why-isc-dhcp-is-actually-useful"></span>
== Why ISC DHCP Is Actually Useful ==

# '''Hostname Resolution''': Use hostnames instead of memorizing IP addresses.
# '''Works with DNS Resolver''': Registers DHCP stuff '''''automatically!''''' You know, like it should.
# '''Simplifies Things''': Makes managing your network a lot easier.

<span id="setting-up-isc-dhcp-in-pfsense"></span>
== Setting Up ISC DHCP in pfSense ==

<span id="make-sure-youre-using-isc-dhcp"></span>
=== 1. Make Sure You’re Using ISC DHCP ===

# Log into '''pfSense'''.
# Go to <code>System</code> &gt; <code>Advanced</code> &gt; <code>Networking</code>.
# In '''DHCP Server''', select '''ISC DHCP'''.
# If it complains about deprecation, just ignore it. Click the checkbox to ignore the annoying warning.
# Hit “Save”.

<span id="configure-dns-resolver"></span>
=== 2. Configure DNS Resolver ===

# Go to <code>Services</code> &gt; <code>DNS Resolver</code>.
# Check these boxes:
#* ☒ “Register DHCP leases in the DNS Resolver”
#* ☒ “Register DHCP static mappings in the DNS Resolver”
#* ☒ “Register connected OpenVPN clients in the DNS Resolver”
# Save and apply changes.

<span id="set-your-domain"></span>
=== 3. Set Your Domain ===

# Navigate to '''System &gt; General Setup'''.
# Set your “Domain” (like “home.arpa” or “local”).
# Save if you made changes.

This setup lets you use hostnames for all your devices, static IPs, and even VPN clients. It’s simple, it works, and it’ll save you a headache later. Sometimes the old way just works.

Static mappings make sure that this IP address of <code>192.168.5.2</code> is reserved for this computer to connect to, so that no other device can take it ''(unless they are spoofing MAC addresses but if someone inside your house is doing just to mess with you, you have bigger problems, that likely end in them getting punched in the mouth).''

<blockquote>'''NOTE:''' Static IP mappings aren’t a big deal when you have a few phones &amp; game consoles attached to your network at home. IF you are running a server, you are running something where clients(aka other phones/computers) are going to want to consistently know where to access it.

Think of your server like your favorite store. When you visit, do you want to have to look through maps to figure out what address they changed to that day? You could… but… wouldn’t it be better if they were at the same place each time you needed to go?
</blockquote>
'''You’ve now set up an Ubuntu Server with a redundant, encrypted secure storage configuration. This setup gives you:'''

* Boot drive redundancy with RAID 1
* Flexibility for future storage management with LVM – we can resize this later if we want to get a bigger drive setup.
* Enhanced security with full-disk encryption (except <code>/boot</code> and <code>/boot/efi</code>).
* There are the ''“uhm akshually”'' people who will say that if you don’t encrypt boot there’s no point in all of this… ''just shut up.''

<span id="why-i-used-virtual-machines-instead-of-docker-for-some-parts-of-my-system"></span>

= Why I Used Virtual Machines Instead of Docker for Some Parts of My System =

<span id="feel-free-to-skip-this-section-scroll-down-to-understanding-the-basics-of-docker-section"></span>
== FEEL FREE TO SKIP THIS SECTION &amp; SCROLL DOWN TO ''“Understanding the basics of Docker”'' section ==

'''This is another section that is completely unnecessary to read if you simply want to get to a working system. Feel free to fast forward to the ''“Understanding the basics of Docker”''. This is here to provide insight into why I structured the guide the way I did.'''

Docker is a great way to managing programs in lightweight, isolated environments. It changed how sysadmins deploy and maintain their systems. Virtual machines are going out of fashion for many sysadmins; but in this guide, you’ll notice that I’ve grouped certain services into '''virtual machines ''' instead of using Docker for everything. Let me explain why

<span id="building-my-system-piece-by-piece"></span>
== 1. Building My System Piece by Piece ==

Back when I started getting into self managing my own servers 15 years ago, my setup wasn’t built all at once. It was cobbled together using the hardware I had lying around; old laptops, physical servers &amp; spare drives. As these machines aged, broke, or were [https://www.youtube.com/watch?v=X86F1j5gCQs bashed in with a titanium nightstick when they frustrated me], I started turning their hard drives into virtual machines. This was as simple as using <code>ddrescue</code> to create an image of the working hard drive, then using Virtual Machine Manager to run that image.

* '''Why not Docker?''' By the time docker even came out(around 2013), I already had 3 virtual machines that were created from disk images of machines that were running in my closet or my store. At the time, Docker didn’t even exist. Rebuilding everything using docker from scratch once it came out wasn’t an efficient use of time while running a business &amp; [https://www.youtube.com/watch?v=qFVwQCFhKSE wasting most of my spare time fighting my state’s incompetent government].

<span id="time-efficient-migration-from-physical-to-virtual"></span>
== 2. Time-Efficient Migration from Physical to Virtual ==

Taking a physical server and turning it into a virtual machine takes no effort.

# Pull drive out of physical server.
# Run <code>ddrescue -f -d -r 3 /dev/sdb phonesystem.iso</code>
# Open virtual machine manager
# Enter a few commands in terminal to create a bridge network interface so the virtual machines work; once done, I never have to do this again for any other virtual machine.
# Import the <code>phonesystem.iso</code> file as a virtual machine.
# Mess with BIOS/UEFI settings if necessary in virtual machine manager to get it to work.
# Assign the virtual machine the amount of CPU cores/RAM I think it should have based on what it is doing.
# Run it.
# Be happy.

It only takes a few seconds to type the commands &amp; click the icons necessary in virtual machine manager. Compare that to re-architecting the entire system for Docker: it would have taken way more effort, downtime, etc. for an improvement in performance I will never notice as a person who has a few users for my server.

Some of these servers were running years before Docker was a thing, &amp; fixing what wasn’t broken made no sense. Virtual machines offered a way to keep my systems running as they were once the hardware died &amp; have them set to back up with no extra work on my part.

Over the years, I changed to using programs that were in docker exclusively. I went from a normal nextcloud deployment where I manually installed everything from scratch(including dependencies) to immich for images which was all docker. I went from self-managed email where the individual components(mailcot, dovecow, mysql, rspamd, etc.) were all installed from scratch to mailcow. Along the way, I just installed the dockerized version of the program on the virtual machine that was assigned to that “group” of services

<span id="certain-programs-arent-built-for-docker"></span>
== 3. Certain Programs Aren’t Built for Docker ==

Not every program is Docker-friendly. For example:

* '''FreePBX (or PBXinaflash):''' You ''could'' theoretically create a custom Docker setup for these, but it would involve a ridiculous amount of work with little to no benefit for someone with my number of users(1, me, or 1 or 2 other people sometimes). For my use case (&amp; most home users), the performance penalty of using FreePBX in a virtual machine instead of dockerized is as noticeable as using '''Gentoo Linux''' with <code>emerge</code> to compile the entire thing from a stage 1 tarball vs. using gentoo with <code>apt</code> to install programs.
* '''Home Assistant:''' The developers themselves recommend using their pre-built VM image (HaOS) instead of Docker. If the devs think it’s better, I’m not going to argue. I can barely write a zfs health monitoring email script. Who am I to argue with the developers of the best home automation software on earth on the best way to run their program?

Even if I wanted to use Docker for everything, I’d still have at least two VMs running. Adding another one or two doesn’t bother me.

<span id="idiotproof-backups---the-most-important-one"></span>
== 4. Idiotproof backups - the most important one ==

This is a beginner’s guide. Backing up Docker volumes, containers, networks, images, and configs is 100% doable. But let’s be honest, it requires some degree of competence. Backing up a virtual machine, on the other hand, is '''completely idiot-proof.'''

* A virtual machine backup is just a single <code>.qcow2</code> disk image and a single <code>.xml</code> configuration file. Drag and drop those two files to another system, import them into Virtual Machine Manager, and the virtual machine runs.
* There’s no need to rebuild Docker containers, recreate volumes, or edit <code>docker-compose.yml</code> files. It’s so simple that someone with absolutely no technical expertise could do it in one click.
* Infact it is so easy that even I can do it. If I can do it, it is truly idiotproof.

The backup script I provide is one I use myself. Once a week, it backs up all of my virtual machines as well as their configuration files to a ZFS pool which will continue running even if several hard drives fail. If I screw something up, it is two terminal commands or a few clicks in the GUI &amp; I’m back up &amp; running as if nothing stupid ever happened.

'''For beginners, when it comes to backups, simplicity is priceless.'''

Added complexity means you are less likely to ''use'' your backup system &amp; less likely to ''understand'' how restoring from a backup works.

<span id="why-this-guide-uses-virtual-machines"></span>
== Why this guide uses virtual machines ==

In this guide, I’ve grouped services into virtual machines because it mirrors how I built my system over the past 15 years. This approach makes it easier for total beginners to back up and restore their setups without worrying about the complexities of Docker. Here’s how I’ve organized the VMs:

# '''Android Services:''' Alternatives to Google Drive, Google Photos, and Google Docs.
# '''Identity and Communication Services:''' Alternatives to Gmail, Google Calendar, Google Contacts, and Google Chrome’s password manager.
# '''Phone System:''' FreePBX for managing calls.
# '''Home Automation:''' Home Assistant for smart home management.

<span id="you-do-not-have-to-do-anything-this-way-if-you-dont-want-to."></span>
== You do not have to do anything this way if you don’t want to. ==

You’re welcome to adapt this setup, or not. If you prefer Docker, you can combine many of these services into one host system. However, I still recommend using virtual machines for the following:

* '''FreePBX:''' The extra effort required to make this work in Docker isn’t worth it.
* '''Home Assistant:''' The HaOS image is the easiest and most reliable way to run Home Assistant, as per the developers’ recommendations.

This guide wasn’t about doing everything new - it was about all of you asking how ''I'' had set up ''my'' system. Since my system works for everything under the sun &amp; has continued to for longer than I’ve been allowed to buy alcohol, I keep it going.

<span id="understanding-the-basics-of-docker"></span>
= Understanding the basics of Docker =

<span id="feel-free-to-skip-this-section-scroll-down-to-configuring-our-servers-networking-for-virtual-machines-section"></span>
== FEEL FREE TO SKIP THIS SECTION &amp; SCROLL DOWN TO ''“Configuring Our Server’s Networking for Virtual Machines”'' section ==

'''You do not need to read this section to install the software in this guide. You can simply copy &amp; paste along commands as I provide them to you, or follow the documentation from the program’s developers. This section is not required reading, but rather here to help you understand the ''how'' and the ''why'' behind the installation methods for the programs we’re installing so you learn as you go - ''if you’re interested.'' If not, skip ahead to ''“Configuring Our Server’s Networking for Virtual Machines”'' '''

We are going to use docker to install a program called '''mailcow'''. Before getting started installing mailcow, I want to go over what docker is &amp; how it works.

'''You do not need to be a genius linux sysadmin at creating your own docker containers &amp; setups to use it, but you should have some clue what it is or what happens when you type <code>docker compose up</code> to run something!''' Docker massively changed how sysadmins run &amp; deploy software. It’s the engine behind many modern self-hosted solutions like ''Mailcow'', ''Immich'', ''Bitwarden'', ''Frigate'', &amp; ''OnlyOffice''. It gets rid of one of the single largest pain points of releasing''(or using, or installing)'' software on Linux: dependencies. Before getting into what Docker is, let’s go over dependency hell.

[http://www.mandrake.tips.4.free.fr/review2006.html <gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:image-20241202004656889.png
</gallery>]

<span id="what-are-dependencies-and-why-do-they-cause-problems"></span>
== What Are Dependencies and Why Do They Cause Problems? ==

<span id="understanding-dependencies"></span>
=== Understanding Dependencies ===

A '''dependency''' is software/libraries/frameworks that have to be installed for the program you are installing to work. Let’s say you’re installing a web application written in PHP; it might need a specific PHP module or a specific version of PHP.

* If you don’t have that version of PHP installed, the application won’t work.
* If you don’t have PHP installed, the application won’t work.
* If you want to use an application that requires a different version of PHP on the same machine….

and so on &amp; so forth.

<div class="figure">

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:800px-Netherlandwarf.jpg
</gallery>

</div>
<span id="the-dependency-hell-of-the-1990s"></span>
=== The Dependency Hell of the 1990s ===

Before modern package managers like <code>apt</code> used by Debian(and 6+ years later, ubuntu) or <code>emerge</code> (Gentoo), installing software on GNU/Linux would require '''manually finding &amp; installing specific dependencies.''' Here’s what this hell was like:

# You downloaded a <code>.tar.gz</code> file that was the source code of the program you wanted to install, called <code>rabbitholetohell</code>.
# You ran <code>./configure</code> &amp; it told you you’re missing <code>libshit</code>.
# You found<code>libshit</code>, downloaded it, and discovered ''it'' required (<code>libpiss</code>).
# You found <code>libpiss</code> but learned that <code>libpiss</code> needed version 1.2 of <code>libpuke</code> and your computer had version 1.3 of <code>libpuke</code> installed.
# Downgrading from version 1.3 of <code>libpuke</code> to version 1.2 of <code>libpuke</code> breaks your entire system.
# User throws keyboard at wall &amp; switches back to windows and says forget GNU/Linux for life.
# If the user is a sysadmin, they curse and figure out how to make it work because this is their job, wasting tons of time.

This was called '''dependency hell''', where each dependency needed more dependencies. it’s what eli the computer guy would correctly call the [https://www.youtube.com/watch?v=I-N_iQC1Uhk rabbit hole to hell]

Tools like <code>apt</code> came along in the late 90s. Instead of dependency hell, you typed <code>apt install rabbitholetohell -y</code> &amp; it just installed <code>rabbitholetohell</code>. It installed all the dependencies, &amp; their dependencies, and it installed the right ones. It was beautiful…

Yet, even with tools like <code>apt</code> to make installs simpler, problems came up if multiple applications needed '''different versions''' of the same dependency. For example:

* '''PHP Example:''' Suppose you wanted to run two applications:
** App 1 requires PHP 7.4.
** App 2 requires PHP 8.1.
** Your system can only have one version of PHP installed at a time, and switching between versions was a [https://www.youtube.com/watch?v=I-N_iQC1Uhk rabbit hole to hell]

<div class="figure">

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:800px-Netherlandwarf.jpg
</gallery>

</div>
<span id="why-this-is-a-nightmare-for-software-maintenance"></span>
=== Why This Is a Nightmare for Software Maintenance ===

Dependencies can become a serious problem over time:

# '''Conflicting Requirements:''' If program A needs <code>libshit</code> version 1.2 &amp; program B needs <code>libshit</code> version 2.0, your system can break when one application upgrades.
# '''Complex Upgrades:''' Updating dependencies for one application can &amp; will cause another application to stop working. This is called '''dependency breakage''' and they are another common cause of chasing rabbits all the way to hell.
# '''System Decay:''' Over time, manually managing dependencies can lead to a bloated, unstable system full of broken packages, outdated libraries, &amp; leftover files.
# '''Version pinning misery:''' <code>apt</code> lets you install specific versions of packages but managing version conflicts becomes timewasting, dangerous, &amp; difficult when dependencies span dozens of packages with intricate relationships. As a newbie, you are likely going to break your system. As an experienced sysadmin… they still broke their systems….

<span id="how-docker-solves-this-mess"></span>
=== How docker solves this mess ===

Docker containers solve these problems by '''isolating dependencies for each application.''' Here’s how it works:

# '''Per-Application Environments:''' Each Docker container includes everything an application needs to run from the application code, runtime, &amp; all dependencies. These are packaged together in the Docker '''image'''.
#* Example: If one application needs PHP 7.4 and another needs PHP 8.1, you can run both simultaneously in separate containers without conflict, on the same computer.
#* I am not talking about on separate virtual machines. I mean on the SAME HOST OPERATING SYSTEM. Two versions of PHP; or ten if you wanted. and no issues. no conflicts. No rabbit, &amp; no hell :)
# '''Immutable(unchangeable):''' Docker images are immutable snapshots. Once built, the dependencies in an image don’t change, so the application runs consistently every time. It’s not like an operating system update where package A may not be updated but package B is, and package A depends on a specific version of package B so everything breaks.
# '''No System-Wide Conflicts:''' Docker containers don't mess with each other on the host system. The PHP version inside the container for <code>nextcloud</code>doesn’t affect the PHP version on the host, or in the container for <code>magento</code>.
# '''Simple Upgrades:''' If you need to update an application you just type <code>docker compose pull</code> when it’s not running &amp; it just updates… seamlessly. If it fails or the dev messed something up, you can go back to a previously installed image without messing up other applications.
# '''Portable:''' Docker makes sure that the program &amp; its dependencies work the same way on ANY system; whether it’s your personal server, a cloud provider, or your friend’s gaming PC.

<span id="why-docker-has-exploded-in-popularity-for-small-open-source-projects"></span>

== Why docker has exploded in popularity for small open source projects ==

<span id="developers-get-less-complaints-from-users"></span>
=== Developers get less complaints from users ===

The biggest complaint by far that many open source projects get is ''I tried to install abc program &amp; get xyz error.'' It is the bane of open source software developer’s existence, until they stop caring about their users entirely. This is often the only way to stay sane in a world where ''“users”''(NOT ''“customers”''), who pay the developers $0, expect unlimited tech support &amp; handholding as well as a one year lesson in GNU/Linux systems administration so they can install a photo gallery.

This sucks.

With Docker, for a developer to hand off a program running on their server to your server, the dev only has to provide you the following:

# Docker image of your application
# The associated Docker Compose <code>docker-compose.yml</code> file
# Instructions or files to set up storage &amp; networking.
# If you want to copy the files over that the service was saving that are unique to you, the docker volume.
# Tell you to edit xyz content in a <code>docker-compose.yml</code> file so the software is set to your specific need.
# Tell you to type <code>docker compose pull</code> &amp; <code>docker compose up -d</code>
# Never hear complaints from you again.

The Docker image contains the program &amp; its environment, which makes sure the software runs the same on their server as it does on yours.

'''AKA, the developers can provide me a COMPLETELY IDIOTPROOF copy of their software that is so easy to install even I can’t screw it up. Once they get it to install on THEIR system - they know it’ll install on mine.'''

The docker-compose.yml file explains to docker &amp; your computer how to “deploy” the program &amp; has details about Docker networks (e.g., how the containers communicate) &amp; Docker volumes (for storing data that persists outside the container).

<span id="docker-makes-what-used-to-be-miserable-very-easy"></span>

=== Docker makes what used to be miserable very easy ===

* You can run '''Mailcow''' (which uses PHP 7.4 for its web interface) alongside '''OnlyOffice''' (which needs PHP 8.1) on the '''same server without issues.'''
* When setting up something like '''Immich''', you don’t need to worry about Node.js versions conflicting with other apps. The devs use Docker to bundle the correct version for you. You don’t have to RTFM to figure out the right version of <code>libshit</code> to install anymore - the developer does that once, and then it’s set for all of their users.
* If a new version of '''Bitwarden''' requires updated dependencies, you update the Docker container, leaving the rest of your system untouched.

Docker turns what used to be a nightmare into a manageable, predictable process that isn’t miserable.

<span id="how-docker-works"></span>
== 1. How Docker Works ==

Docker simplifies running software by packaging everything the software needs into one neat bundle. It does this using '''containers''' which are lightweight standalone environments that share the host system’s resources but remain isolated.

This is like a virtual machine, but without the baggage of needing its own operating system. Docker containers run on a shared kernel, making them much faster and lighter. If you ever enter a docker container, you will notice that almost no programs or commands are available besides the ''bare minimum'' necessary to do the job. See below:

<pre>louis@ultimatebauer:~$ docker exec -it frigate bash
root@174eb3845d50:/opt/frigate# nano file.log
bash: nano: command not found
root@174eb3845d50:/opt/frigate# vi file.log
bash: vi: command not found
root@174eb3845d50:/opt/frigate# vim file.log
bash: vim: command not found
root@174eb3845d50:/opt/frigate# emacs file.log
bash: emacs: command not found
root@174eb3845d50:/opt/frigate# ip addr show
bash: ip: command not found
root@174eb3845d50:/opt/frigate#  you really don't have shit in here besides exactly what you need to run the application, do you? run nano you prick!</pre>
<blockquote>root@174eb3845d50:/opt/frigate '''I’m afraid I can’t do that, dave'''
</blockquote>
<span id="what-are-docker-images"></span>
== 2. What Are Docker Images? ==

A '''Docker image''' is a blueprint on how to install the program. It has the instructions, files, &amp; dependencies necessary to create a working environment for a piece of software. Think of it like a frozen dinner if they weren’t poisonous to your health. Everything you need is pre-packaged, &amp; all you have to do is microwave it ''(or, in this case, “run” the image; please don’t try to microwave a GNU/Linux computer, as tempting as it might be when it doesn’t work)'' to get the app running.

* Example: A <code>Nextcloud</code> Docker image includes the Nextcloud app, its web server, and everything else it needs to limp. I won’t use the word ''“run”'' to describe nextcloud…

<span id="what-are-docker-containers"></span>
== 3. What Are Docker Containers? ==

A '''Docker container''' is a running instance of a Docker image. Using the frozen dinner analogy, if the image is a boxed meal in a freezer, a container is a meal served hot &amp; ready to eat. You can run many containers from the same image just like you could cook multiple identical dinners from the same recipe.

For instance, mailcow is not a mail ''“program”'' so much as it is an amalgamation of a bunch of programs necessary to run a mailserver.

On my mailserver, you can see a list of all the different containers that run for mailcow when I run <code>docker ps -a</code>

<span id="example-mailcow-container-guide"></span>
=== Example: mailcow container guide ===

<span id="mail-processing"></span>
==== Mail processing ====

* '''postfix''': The program that sends emails
* '''dovecot''': The program that receives emails &amp; stores them &amp; categorizes them by user, inbox, email address, folder, etc.
* '''rspamd''': anti-spam controls
* '''clamd''': scans attachments for viruses

<span id="web-interface"></span>
==== Web &amp; Interface ====

* '''sogo''': webmail dashboard for checking email/calendar/contacts in browser
* '''phpfpm''': for web interface

<span id="security-monitoring"></span>
==== security &amp; monitoring ====

* '''watchdog''': The health monitor
* '''acme''': Handles SSL certificates
* '''netfilter''': Blocks bad actors
* '''unbound''': helps route messages correctly

<span id="helper-services"></span>
==== Helper Services ====

* '''solr''': Makes searching through your email faster
* '''olefy'''
* '''dockerapi''':

Think of Docker containers like having separate tiny computers inside your main computer that are barebones and only include the minimum necessary for each function to work. They each work independent of each other to minimize the likelihood of something screwing up while also allowing you the ability to experiment without destroying your entire system.

Containers are not '''persistent.''' This means what happens in the containers stays in the container until you restart it. Once you restart the container, any changes to files you have made are ''GONE''. '''PERSISTENT''' storage occurs in docker ''volumes''.

Each container has its own:

* Space to run programs
* Network connection
* File storage
* Settings
* Installed programs

Unlike full virtual machines (which are like having complete separate computers), containers share the main operating system’s foundation ''(the host’s operating system kernel)'', making them much lighter and faster to start up.

For example, in mailcow:

* The postfix container only knows about sending/receiving mail
* The rspamd container is only for filtering junk
* The clamd container is only there to scan for viruses

They can’t interfere with each other, but they can communicate through specific “doorways” (network ports) when needed. If something goes wrong with one container, it doesn’t affect the others - just like one apartment’s plumbing problem doesn’t affect the other apartments (hopefully).

If you need to upgrade or fix something, you can work on one container without messing with everything else.

<pre>louis@mailserver:~$ docker ps -a
CONTAINER ID   IMAGE                    COMMAND                  CREATED       STATUS                  PORTS                                                                                                                                                                                                                               NAMES
aca88eab00b0   mailcow/watchdog:2.05    &quot;/watchdog.sh&quot;           11 days ago   Up 24 hours                                                                                                                                                                                                                                                 mailcowdockerized-watchdog-mailcow-1
012debb1f557   mailcow/acme:1.90        &quot;/sbin/tini -g -- /s…&quot;   11 days ago   Up 24 hours                                                                                                                                                                                                                                                 mailcowdockerized-acme-mailcow-1
d33aa2bb976b   nginx:mainline-alpine    &quot;/docker-entrypoint.…&quot;   11 days ago   Up 24 hours             0.0.0.0:80-&gt;80/tcp, :::80-&gt;80/tcp, 0.0.0.0:443-&gt;443/tcp, :::443-&gt;443/tcp                                                                                                                                                            mailcowdockerized-nginx-mailcow-1
7bc85825c0b1   mailcow/rspamd:1.98      &quot;/docker-entrypoint.…&quot;   11 days ago   Up 24 hours                                                                                                                                                                                                                                                 mailcowdockerized-rspamd-mailcow-1
958d3ba45877   mcuadros/ofelia:latest   &quot;/usr/bin/ofelia dae…&quot;   11 days ago   Up 24 hours                                                                                                                                                                                                                                                 mailcowdockerized-ofelia-mailcow-1
a99f82d2b36a   mailcow/phpfpm:1.91.1    &quot;/docker-entrypoint.…&quot;   11 days ago   Up 24 hours             9000/tcp                                                                                                                                                                                                                            mailcowdockerized-php-fpm-mailcow-1
b8c6df6a7303   mailcow/dovecot:2.2      &quot;/docker-entrypoint.…&quot;   11 days ago   Up 24 hours             0.0.0.0:110-&gt;110/tcp, :::110-&gt;110/tcp, 0.0.0.0:143-&gt;143/tcp, :::143-&gt;143/tcp, 0.0.0.0:993-&gt;993/tcp, :::993-&gt;993/tcp, 0.0.0.0:995-&gt;995/tcp, :::995-&gt;995/tcp, 0.0.0.0:4190-&gt;4190/tcp, :::4190-&gt;4190/tcp, 127.0.0.1:19991-&gt;12345/tcp   mailcowdockerized-dovecot-mailcow-1
e3b09c799a7c   mailcow/postfix:1.77     &quot;/docker-entrypoint.…&quot;   11 days ago   Up 24 hours             0.0.0.0:25-&gt;25/tcp, :::25-&gt;25/tcp, 0.0.0.0:465-&gt;465/tcp, :::465-&gt;465/tcp, 0.0.0.0:587-&gt;587/tcp, :::587-&gt;587/tcp, 588/tcp                                                                                                            mailcowdockerized-postfix-mailcow-1
faece81357e3   mailcow/solr:1.8.3       &quot;docker-entrypoint.s…&quot;   11 days ago   Up 24 hours             127.0.0.1:18983-&gt;8983/tcp                                                                                                                                                                                                           mailcowdockerized-solr-mailcow-1
76c9f63fa50d   mariadb:10.5             &quot;docker-entrypoint.s…&quot;   11 days ago   Up 24 hours             127.0.0.1:13306-&gt;3306/tcp                                                                                                                                                                                                           mailcowdockerized-mysql-mailcow-1
930a7e0acff6   redis:7-alpine           &quot;docker-entrypoint.s…&quot;   11 days ago   Up 24 hours             127.0.0.1:7654-&gt;6379/tcp                                                                                                                                                                                                            mailcowdockerized-redis-mailcow-1
8bbcbe5ebefb   mailcow/clamd:1.66       &quot;/sbin/tini -g -- /c…&quot;   11 days ago   Up 24 hours (healthy)                                                                                                                                                                                                                                       mailcowdockerized-clamd-mailcow-1
9070a5ba3fb0   mailcow/olefy:1.13       &quot;python3 -u /app/ole…&quot;   11 days ago   Up 24 hours                                                                                                                                                                                                                                                 mailcowdockerized-olefy-mailcow-1
893f2ff1f952   mailcow/dockerapi:2.09   &quot;/bin/sh /app/docker…&quot;   11 days ago   Up 24 hours                                                                                                                                                                                                                                                 mailcowdockerized-dockerapi-mailcow-1
6781988f3409   mailcow/sogo:1.127.1     &quot;/docker-entrypoint.…&quot;   11 days ago   Up 24 hours                                                                                                                                                                                                                                                 mailcowdockerized-sogo-mailcow-1
464ca438b4c2   mailcow/unbound:1.23     &quot;/docker-entrypoint.…&quot;   11 days ago   Up 24 hours (healthy)   53/tcp, 53/udp                                                                                                                                                                                                                      mailcowdockerized-unbound-mailcow-1
373c1b7c5741   mailcow/netfilter:1.59   &quot;/bin/sh -c /app/doc…&quot;   11 days ago   Up 24 hours                                                                                                                                                                                                                                                 mailcowdockerized-netfilter-mailcow-1
6931fc976572   memcached:alpine         &quot;docker-entrypoint.s…&quot;   11 days ago   Up 24 hours             11211/tcp                                                                                                                                                                                                                           mailcowdockerized-memcached-mailcow-1
louis@mailserver:~$ </pre>
<span id="what-are-docker-networks"></span>

== 4. What Are Docker Networks? ==

Docker allows containers to communicate with each other &amp; the outside world using '''networks'''. By default, the containers can access the internet. Custom networks allow you to connect certain containers while keeping them separate from others.

For instance, in '''mailcow''' docker networks make sure the mail server can talk to the database container securely without exposing the database to the entire internet.

<span id="what-are-docker-volumes"></span>
== 5. What Are Docker Volumes? ==

A '''Docker volume''' is where data generated by a container is stored. Think of a docker container like a computer booting up from a read only floppy disk. Whatever you ran in your programs is gone the second you reboot the computer. The docker volume is the second disk in the computer that you can write to so that you can save things. Containers are where programs run (postfix, dovecot), and volumes are where things are stored (emails, pictures, videos, etc.). Volumes make sure that important data persists even if the container is removed or restarted.

<span id="volume-examples-with-different-programs"></span>
=== Volume examples with different programs: ===

The <code>docker-compose.yml</code> file is what tells docker how to set up everything. In frigate, we are not creating docker volumes. Rather, we tell docker to map a directory on the host computer inside the docker container. Look here:

<span id="docker-program-that-does-not-use-docker-volumes"></span>
==== docker program that does not use docker volumes ====

In this file, the container '''“frigate”''' specified on line 4 by ''container_name'', we do not have any docker volumes specified. Under <code>services</code> we specify our containers. There are no docker volumes specified here. We have told the system that whatever is in <code>/home/louis/Downloads/programs/frigate/config</code> on the host system should show up inside the <code>frigate</code> container on the directory <code>/config</code>. Without this, the <code>config.yml</code> file within the <code>/home/louis/Downloads/programs/frigate/config</code> directory would not show up inside the container. Even if I logged into the container using <code>docker exec -it frigate bash</code> and created a <code>config.yml</code> file in <code>/config</code>, it would be gone when I restarted the container.

<pre>version: &quot;3.9&quot;
services:
  frigate:
    container_name: frigate
    privileged: true # this may not be necessary for all setups
    restart: unless-stopped
    image: ghcr.io/blakeblackshear/frigate:stable
    shm_size: &quot;2048mb&quot; # update for your cameras based on calculation above
    devices:
      - /dev/bus/usb:/dev/bus/usb # Passes the USB Coral, needs to be modified for other versions
      - /dev/apex_0:/dev/apex_0 # Passes a PCIe Coral, follow driver instructions here https://coral.ai/doc&gt;
      - /dev/video11:/dev/video11 # For Raspberry Pi 4B
      - /dev/dri/renderD128:/dev/dri/renderD128 # For intel hwaccel, needs to be updated for your hardware
    volumes:
      - /etc/localtime:/etc/localtime:ro
      - /home/louis/Downloads/programs/frigate/config:/config
      - /drive1thru8/securitycam:/data/db
      - /drive1thru8/securitycam:/media/frigate
      - type: tmpfs # Optional: 1GB of memory, reduces SSD/SD Card wear
        target: /tmp/cache
        tmpfs:
          size: 1000000000
    ports:
      - &quot;8971:8971&quot;
      - &quot;5000:5000&quot; # Internal unauthenticated access. Expose carefully.
      - &quot;8554:8554&quot; # RTSP feeds
      - &quot;8555:8555/tcp&quot; # WebRTC over tcp
      - &quot;8555:8555/udp&quot; # WebRTC over udp
    environment:
      FRIGATE_RTSP_PASSWORD: &quot;password&quot;</pre>
<span id="docker-program-that-does-use-docker-volumes"></span>
==== docker program that DOES use docker volumes ====

Check out mailcow. This is not the full <code>docker-compose.yml</code> configuration file, just a part of it. Look at lines 25-28. For the container <code>mysql-mailcow</code>, we have two docker volumes. The docker volume <code>mysql-vol-1</code> will show up inside the <code>mysql-mailcow</code> container'' (which is a tiny virtual computer that runs our programs, in this case, the mysql database. mysql databases usually contain data on users, configurations, product orders, etc)''. Whatever is in the <code>mysql-vol-1</code> docker volume will show up inside the <code>mysql-mailcow</code> container at <code>/var/lib/mysql</code>.

It is using a docker volume instead of the main computer/operating system’s file system to store its files.

However, on line 28, we have <code>- ./data/conf/mysql/:/etc/mysql/conf.d/:ro,Z</code> which means that whatever is in the subfolder of our mailcow folder''(where the <code>docker-compose.yml</code> file is that we used to install mailcow)'' under <code>data/conf/mysql/</code> will show up inside the docker container at <code>/etc/mysql/conf.d/</code>

<pre>services:

    unbound-mailcow:
      image: mailcow/unbound:1.23
      environment:
        - TZ=${TZ}
        - SKIP_UNBOUND_HEALTHCHECK=${SKIP_UNBOUND_HEALTHCHECK:-n}
      volumes:
        - ./data/hooks/unbound:/hooks:Z
        - ./data/conf/unbound/unbound.conf:/etc/unbound/unbound.conf:ro,Z
      restart: always
      tty: true
      networks:
        mailcow-network:
          ipv4_address: ${IPV4_NETWORK:-172.22.1}.254
          aliases:
            - unbound

    mysql-mailcow:
      image: mariadb:10.5
      depends_on:
        - unbound-mailcow
        - netfilter-mailcow
      stop_grace_period: 45s
      volumes:
        - mysql-vol-1:/var/lib/mysql/
        - mysql-socket-vol-1:/var/run/mysqld/
        - ./data/conf/mysql/:/etc/mysql/conf.d/:ro,Z
      environment:
        - TZ=${TZ}
        - MYSQL_ROOT_PASSWORD=${DBROOT}
        - MYSQL_DATABASE=${DBNAME}
        - MYSQL_USER=${DBUSER}
        - MYSQL_PASSWORD=${DBPASS}
        - MYSQL_INITDB_SKIP_TZINFO=1
      restart: always
      ports:
        - &quot;${SQL_PORT:-127.0.0.1:13306}:3306&quot;
      networks:
        mailcow-network:
          aliases:
            - mysql</pre>
<span id="mailcow-docker-volume-descriptions"></span>
===== mailcow docker volume descriptions =====

Here are some docker volumes used for mailcow:

<pre>louis@mailserver:/opt/mailcow-dockerized$ docker volume ls
DRIVER    VOLUME NAME
local     mailcowdockerized_clamd-db-vol-1
local     mailcowdockerized_crypt-vol-1
local     mailcowdockerized_mysql-socket-vol-1
local     mailcowdockerized_mysql-vol-1
local     mailcowdockerized_postfix-vol-1
local     mailcowdockerized_redis-vol-1
local     mailcowdockerized_rspamd-vol-1
local     mailcowdockerized_sogo-userdata-backup-vol-1
local     mailcowdockerized_sogo-web-vol-1
local     mailcowdockerized_solr-vol-1
local     mailcowdockerized_vmail-index-vol-1
local     mailcowdockerized_vmail-vol-1
</pre>
<span id="main-data-storage"></span>
====== main data storage ======

* <code>vmail-vol-1</code>: The emails &amp; attachment files
* <code>mysql-vol-1</code>: Database stuff like user accounts/settings
* <code>redis-vol-1</code>: Temporary data for faster load times

<span id="email-processing"></span>
====== email processing ======

* <code>postfix-vol-1</code>: Mail server configuration &amp; logs
* <code>rspamd-vol-1</code>: spam filter rules &amp; training data
* <code>clamd-db-vol-1</code>: Virus scanning database

<span id="webmail-user-data"></span>
====== webmail &amp; user data ======

* <code>sogo-userdata-backup-vol-1</code>: Backups of user settings &amp; data
* <code>sogo-web-vol-1</code>: Web interface files
* <code>vmail-index-vol-1</code>: Helps search through old email quickly

<span id="random-technical-volumes"></span>
====== random technical volumes ======

* <code>crypt-vol-1</code>: Encryption-related data
* <code>mysql-socket-vol-1</code>: This assists database communication
* <code>solr-vol-1</code>: Search engine data

<span id="this-seems-like-a-lot"></span>
== This seems like a lot ==

If this is too much, realize this. 99% of installing programs that are packaged with docker means doing the following:

# Downloading a <code>docker-compose.yml</code> file
# Running the command <code>docker compose pull</code> to grab program
# Running the command <code>docker compose up -d</code> to start program.
# You’re done.
# If an idiot like me can do it, then so can you.

'''YOU DO NOT NEED TO BECOME AN EXPERT SYSTEMS ADMINISTRATOR OVERNIGHT.'''

The best way to learn is to try and understand things one part at a time. You do it like this:

# Set something up, have it work.
# Have no idea what you did.
# Mess around with it &amp; enjoy it.
# Use the kick of dopamine from it working &amp; enjoying it to get motivated.
# Read a piece of a config file just for the hell of it and see if it maps to anything in the program/what you’re doing.
# If it makes no sense, don’t worry about it, keep enjoying the program &amp; increasing your stock of dopamine &amp; happiness &amp; satisfaction.
# Come back to it again later.
# Read a little bit.
# Read something on a forum/manual/guide that makes little sense to you, but maybe 1% more sense now than it did a week ago.
# Pat yourself on the back for understanding it even though you think this is kindergarten level &amp; you’re an idiot &amp; everyone else knows way more than you.
# Enjoy program more.
# Don’t crap on yourself because you don’t get everything.
# When bored sitting in a meeting you have no business wasting your time in, alt-tab over to your <code>docker-compose.yml</code> file.
# Google random parts &amp; see what they do.
# Think about how that piece of software works. Google what the different words inside of the file do, what those programs are for, &amp; how they relate to the program working as a whole.
# See if you understand 1% more now than before.
# Each percent you understand is not '''cumulative''' - it is '''''exponential!''''' Learning this stuff is a parabola. In the beginning, it is insanely slow. Once you get started &amp; understand the foundation, learning increases at an exponential pace.
# You need to overcome that period where you feel like an imposter &amp; a total idiot in order to get better.
# Realize that even complete experts know [https://wiki.futo.org/index.php/FUTO:General_disclaimer 0.0001%] of what there is to know about all of this and usually specialize in one specific area, because to understand how everything works is damn near impossible.

<span id="configuring-our-servers-networking-for-virtual-machines"></span>

= Configuring Our Server’s Networking for Virtual Machines =

<span id="what-are-virtual-machines"></span>
== What are virtual machines? ==

We are going to make use of virtual machines a lot. Virtual machines (VMs) are software-based computers running inside your physical server. This approach allows us to have separated, segmented computers running inside of our computer that are '''absolutely idiotproof to back up &amp; restore.''' Key word there being ''idiotproof.'' Once I provide you with a working backup script, if you mess something up with any of the services (mailcow, freepbx, homeassistant syncthing, immich, nextcloud, etc.) all you have to do is:

# Shut down the existing messed up virtual machines.
# Restore a single <code>.qcow2</code> file from backup.
# Start up the virtual machine.

And everything works again. No confusing command line stuff, no editing config files. Depending on the host &amp; network you move it to, you might have to edit the IP address configuration &amp; ports forwarded in the firewall; besides that it will just '''work.''' This is beautiful, and so idiotproof even someone like me can do it. If I mess up my phone system, I can restore it in seconds without having to mess with any other part of my system. Did I mention it’s idiotproof? This is the most important quality of a system when I’m the one using it.

'''If the server’s hardware or OS drive fails, it’s easy to move the VMs to another device. Insanely easy. Take a backup of a single file &amp; move it over easy.'''

<blockquote>'''''nerd note:''' Yes, docker allows for containerized installs of everything. Yes, it’s faster, yes, it makes more sense in an enterprise environment… this is a beginner’s guide. Having a very easy backup script that allows copying &amp; pasting a qcow2 file when you break something means you’ll actually '''''use''''' your backups rather than give up, which is important in the beginning. We will create segmented VMs for each of our purposes (identity/email, android/cloud services &amp; sync, home automation, etc.) and they will have programs &amp; services running in docker within them; often several. The backup solution will be backing up these VMs because it is dirt easy for a beginner vs. managing backing up all associated docker containers &amp; volumes. If you want to manage this, go set up Kubernetes at a mid-sized company for someone &amp; stop reading a newbie guide.''
</blockquote>
By running each set of services in its own VM, we isolate them from each other—so if one service has an issue, it won’t bring down the entire system.

'''One problem: I need these virtual machines to connect to the internet… and since they’re virtual, they have no network interface card… so I can’t plug them into my switch.'''

Since VMs don’t come with physical network interface cards (NICs) like a regular server, we need to create a virtual network interface that allows them to connect to the network and access the internet. This virtual interface acts as a bridge between the VMs and your server’s physical network connection.

<span id="step-1-disable-cloud-inits-network-configuration"></span>
=== Step 1: Disable Cloud-Init’s Network Configuration ===

Before changing the network configuration, you need to stop cloud-init from managing it.

Create the file to disable cloud-init’s network management by running the following command: <code>sudo nano /etc/cloud/cloud.cfg.d/99-disable-network-config.cfg</code> Add this to the file: <code>network: {config: disabled}</code> Save &amp; close the file by typing <code>Ctrl + X</code> and hitting <code>y</code> to save it.

<span id="step-2-backup-the-current-netplan-configuration"></span>
=== Step 2: Backup the Current Netplan Configuration ===

Make a backup of your current Netplan configuration.

Run the following command to back up the current <code>50-cloud-init.yaml</code> file: <code>sudo mv /etc/netplan/50-cloud-init.yaml /etc/netplan/50-cloud-init.yaml.bak</code>

<code>.bak</code> makes sure that Netplan will not use it for creating a configuration. <code>.bak</code> also makes it easy to go “back” – you don’t have to remember the filename or the location of the original file. You just copy the backup file to the same filename in the same directory without the suffix <code>.bak</code>, there it is.

<span id="step-3-create-a-new-netplan-configuration"></span>
=== Step 3: Create a New Netplan Configuration ===

Since you disabled cloud-init, you can now modify the network configuration to create a bridge interface that your virtual machines can use.

# Find the name of your ethernet interface (the one the CAT5 cable plugs into):

<pre>[louis@livingroombauer ~]$ ls /sys/class/net
   enp4s0  lo</pre>
In my personal computer, <code>enp4s0</code> is my network interface, and <code>lo</code> is the '''loopback''' interface. '''loopback''' allows the machine to talk to itself - other computers cannot contact this computer through the loopback interface. This is useful if there is a service we would like to run that we do not want to be accessible to other machines on the network. <code>enp4s0</code> is my ethernet port. On my '''server''', <code>eno1</code> is my interface that the ethernet port plugs into, so I will use that below. When you see me using <code>eno1</code> as I set up my server, replace <code>eno1</code> with the actual name of your network interface.

<ol style="list-style-type: decimal;">
<li><p>Create or edit the Netplan configuration file by running this command:</p>
<pre>sudo nano /etc/netplan/01-netcfg.yaml</pre></li>
<li><p>Replace the content with the following configuration. I’ve added a comment on each line so you know how many spaces there should be:</p>
<pre>network: # 0 spaces
  version: 2 # 4 spaces
  renderer: networkd # 4 spaces
  ethernets: # 4 spaces
    eno1: # 8 spaces
      dhcp4: no # 12 spaces
  bridges: # 4 spaces
    br0: # 8 spaces
      dhcp4: no # 12 spaces
      addresses: # 12 spaces
        - 192.168.5.2/24 # 16 spaces
      nameservers: # 12 spaces
        addresses: # 16 spaces
          - 192.168.5.1 # 20 spaces
      routes: # 12 spaces
        - to: default # 16 spaces
          via: 192.168.5.1 # 18 spaces
      interfaces: # 12 spaces
        - eno1 # 16 spaces</pre></li>
<li><p>Once done, remember to change the permissions of your netplan file so netplan does not yell at you:</p>
<pre>sudo chmod 600 /etc/netplan/01-netcfg.yaml</pre></li></ol>

<blockquote>'''Explanation of the Configuration:''' - <code>eno1</code> will be part of the bridge (<code>br0</code>), but will no longer have an IP address directly. - <code>br0</code> is the bridge interface that will be assigned the static IP <code>192.168.5.2</code>. - The <code>br0</code> interface will be configured with the same gateway and nameserver settings as before. The gateway is our pfSense router, which is what it connects to to get an IP address and connect to the internet (a “gateway” to the world), and the nameserver is also our router, which is what it connects to to translate things like google.com into <code>142.250.138.101</code>. - <code>br0</code> is a virtual interface WE are creating. <code>eno1</code> is an interface already present on this machine. <code>eno1</code> is the ethernet port on my computer, simply put. '''Your network interface card will most likely be called something else; this is ok! Use what your network interface is called as it will be different for all machines'''
</blockquote>
<blockquote>'''NOTE:''' You are probably used to old school configuration files where: <code>pasv_enable=YES</code> is the same as <code>pasv_enable=YES</code> That is not how a YAML do. A single space is all that stands between you having a working setup &amp; happiness, and total misery. YAML is sensitive to spaces; indentation errors matter, and can cause the config file to not work. Some text editors are helpful in editing yaml files so that it is easier to notice mistakes &amp; errors. Some are not.
</blockquote>
<span id="step-4-apply-the-new-configuration"></span>
=== Step 4: Apply the New Configuration ===

Now that the configuration is ready, apply it.

<ol style="list-style-type: decimal;">
<li><p>Run the following command to apply the new Netplan configuration:</p>
<pre>sudo netplan apply</pre>
<blockquote><p>'''NOTE:''' You may make an error because yaml files are evil; to make sure the configuration works, run <code>netplan try</code> before running <code>netplan apply</code>. While yoda had a point with the ''“do or not do there is no try”'', he never dealt with linux documentation.</p></blockquote></li>
<li><p>Verify that the bridge interface is up and has the correct configuration by running this command:</p>
<pre>ip addr show br0</pre></li>
<li><p>You should see that <code>br0</code> has the IP address '''192.168.5.2'''.</p></li>
<li><p>Check the routing table to make sure that the default route is correctly set by running:</p>
<pre>ip route show</pre></li>
<li><p>Verify that the default route points to '''192.168.5.1'''.</p></li></ol>

<span id="step-5-test-network-configuration"></span>
=== Step 5: Test Network Configuration ===

Verify that your server can still access the network after the changes.

<ol style="list-style-type: decimal;">
<li><p>Ping your router by running:</p>
<pre>ping 192.168.5.1</pre></li>
<li><p>Ping an external IP to make sure connectivity by running. This is Google’s DNS server, which should be up all the time:</p>
<pre>ping 8.8.8.8</pre></li></ol>

<span id="step-6-add-iptables-rules-for-bridging"></span>
=== Step 6: Add iptables rules for bridging ===

For the bridge to work correctly, you need to allow traffic forwarding on the <code>br0</code> bridge interface. This requires creating iptables rules &amp; making them persistent across reboots. This is a very important detail, often [https://www.tecmint.com/create-network-bridge-in-ubuntu/ left out of guides on setting up bridge interfaces]. Skipping this part will result in a setup that doesn’t work; you will be stuck in the hell of posting on GNU/Linux forums where people with IQs of 180+ will tell you to ''“RTFM”'', a man page, that is 2000000+ pages long.

This is analogous to Derek Jeter telling you to ''“just keep your eye on the ball.”'' right.

You may wonder why that is the case. Setting up things with open source software is like MacBook board repair 12 years ago: it’s a club &amp; you’re not in it. Most teachers know their subject matter. As a result, they forget what it was like to try something for the first time.

This is why I am building a machine from scratch as I do this. Telling you how I did it on my machine will never work. There will always be some small detail I subconsciously assume you will know; or perhaps a detail I forgot myself since some of these services I’m showing you were set up in my closet ten years ago!

By performing the tasks from what I have written, I am forced to provide you with instructions that '''''actually work!'''''

# Run the following commands to add the iptables rules:

<pre> sudo iptables -I FORWARD 1 -i br0 -j ACCEPT
 sudo iptables -I FORWARD 1 -o br0 -j ACCEPT</pre>
'''NOTE:''' These iptables rules let traffic go through the bridge interface so your virtual machines can work on your network. Without them, your virtual machines will not be able to connect to anything, and you won’t be able to connect to them. If you see that your virtual machine received an IP address in virtual machine manager, but it can’t connect to anything, you likely skipped this step. &gt; &gt;The order of rules in iptables matters. Inserting rules at the top (using the ‘-I’) puts them at the top. If traffic forwarding does not work as expected, check rules &amp; the order which you can do by running ‘sudo iptables -L -v -n’.

<ol start="2" style="list-style-type: decimal;">
<li><p>Verify the iptables rules by running:</p>
<pre>sudo iptables -L</pre>
<p>You should see the rules for accepting traffic on <code>br0</code>.</p></li></ol>

<span id="step-7-make-iptables-rules-persistent"></span>
=== Step 7: Make iptables Rules Persistent ===

To make sure the iptables rules are applied after a reboot, you need to save them and configure them to load automatically on startup.

<ol style="list-style-type: decimal;">
<li><p>Install the iptables-persistent package:</p>
<pre>sudo apt install iptables-persistent</pre></li>
<li><p>During installation, you’ll be asked if you want to save the current iptables rules. Choose '''Yes'''.</p></li>
<li><p>If you’re not prompted, you can manually save the rules by running:</p>
<pre>sudo netfilter-persistent save</pre>
<blockquote><p>'''NOTE:''' Installing <code>iptables-persistent</code> is what allows your iptables rules to stick after a reboot. This is a server - you’re not going to turn this off very often. Nine months from now when you DO turn off this server, you’re not going to remember a single damn character from this guide; much less that iptables rule above! Nor will you remember that that rule not being present is why none of your virtual machines work.</p></blockquote></li>
<li><p>Confirm the rules are saved by checking the file at <code>/etc/iptables/rules.v4</code>.</p></li></ol>

With these changes, your bridge interface will now correctly allow traffic to flow through the virtual machines. The iptables rules will persist across reboots, and your virtual machines will be able to grab IP addresses from the same network as your host machine.

<span id="preparing-ubuntu-server-for-virtual-machine-management"></span>
= Preparing Ubuntu Server for Virtual Machine Management =

Next, let’s set up Ubuntu Server for use with virtual machines using '''Virtual Machine Manager''' (virt-manager). We’ll cover everything from preparing the ISO file to configuring the virtual machine with a static IP address, including the installation of a lightweight GUI for easier management.

<span id="step-1-prepare-the-ubuntu-server-iso"></span>
== Step 1: Prepare the Ubuntu Server ISO ==

Before creating the virtual machine, you need to place the Ubuntu Server ISO file in the correct directory and set the proper permissions.

<ol style="list-style-type: decimal;">
<li><p>Place the ISO file you used to create your installable Ubuntu USB onto your server. You can do this by attaching a disk to it, using SSH FTP and a program like filezilla to transfer it over. Or, if you’re an animal, you can download it again by going to ubuntu.com and downloading the LTS version of Ubuntu server again.</p></li>
<li><p>Move the ISO file to <code>/var/lib/libvirt/images/</code>, obviously changing the source location &amp; filename to whatever yours is. As long as the file ends up in <code>/var/lib/libvirt/images/</code> we’re good:</p>
<pre>sudo mv ~/Downloads/ubuntu-server.iso /var/lib/libvirt/images/</pre></li>
<li><p>Change the ownership and group of the ISO file:</p>
<pre>sudo chown libvirt-qemu:libvirt /var/lib/libvirt/images/ubuntu-server.iso</pre></li>
<li><p>Set the correct permissions:</p>
<pre>sudo chmod 0640 /var/lib/libvirt/images/ubuntu-server.iso</pre></li>
<li><p>To apply these settings to all ISO files in the directory:</p>
<pre>sudo chown libvirt-qemu:libvirt /var/lib/libvirt/images/*.iso
sudo chmod 0640 /var/lib/libvirt/images/*.iso</pre></li></ol>

<blockquote>'''Note''': These settings make sure that the <code>libvirt-qemu</code> user, which runs the QEMU processes, can read and write the file, while members of the <code>libvirt</code> group can read it. Other users will have no access, so <code>virsh</code> &amp; related tools can access the ISO files but others can’t.
</blockquote>
<span id="step-2-update-your-system"></span>
== Step 2: Update Your System ==

Make sure your system is up to date:

<pre>sudo apt update ; sudo apt upgrade -y</pre>
''Note:'' Some GNU/Linux distributions update during installation, but it’s always good to check.

<span id="step-3-install-openbox-and-virtual-machine-manager"></span>
== Step 3: Install Openbox and Virtual Machine Manager ==

We’ll install a lightweight desktop environment (Openbox) and Virtual Machine Manager:

<pre>sudo apt install --no-install-recommends xorg openbox xorg xinit virtualbox virtinst qemu-kvm libvirt-daemon-system libvirt-clients bridge-utils virt-manager</pre>
The <code>--no-install-recommends</code> flag makes sure only the core components are installed without any additional unnecessary packages.

<span id="step-4-enable-and-start-libvirt"></span>
== Step 4: Enable and Start Libvirt ==

Enable libvirt to start on boot and start it immediately:

<pre>sudo systemctl enable libvirtd
sudo systemctl start libvirtd</pre>
<span id="step-5-add-your-user-to-necessary-groups"></span>
== Step 5: Add Your User to Necessary Groups ==

To allow your user to configure virtual machines, add yourself to the required groups:

<pre>sudo usermod -aG libvirt,kvm $USER</pre>
<blockquote>'''NOTE:''' Adding your user to the <code>libvirt</code> &amp; <code>kvm</code> groups is useful so you do not have to become superuser/sudo for <code>virt-manager</code>(virtual machine manager GUI) or<code>virsh</code> to work right. Log out &amp; log back in to make sure you’re in the user group after doing this.
</blockquote>
<span id="step-6-start-the-gui"></span>
== Step 6: Start the GUI ==

To start the graphical interface, use the following command:

<pre>startx</pre>
<blockquote>'''Note:''' We will be using the GUI for Virtual Machine Manager. Any time you are NOT using this, exit the GUI (right click the desktop &amp; log out), then type <code>exit</code> from the command line terminal so your machine is not logged in. Even if someone breaks into your house, they’ll have physical access to your computer; but that doesn’t mean they have easy access to your encrypted data.
</blockquote>
<span id="optional-remote-desktop-access-with-x11vnc-and-tigervnc"></span>
==== 6.1(OPTIONAL): Remote Desktop Access with x11vnc and TigerVNC ====

x11vnc is like teamviewer if teamviewer were open source and 50x slower. x11vnc allows you to connect to your server’s GUI for remote access, as if you were right in front of the computer, without having to be in the room with your computer. Up until now, we have been SSHing into the machine in order to enter terminal commands, but normal SSH won’t work if we want to use the graphical user interface, see a mouse cursor, etc.

<blockquote>'''Note:''' If you are ok with sitting in front of your server computer with a keyboard, mouse, &amp; monitor plugged into it when using virtual machine manager, this step is ''unnecessary'' and you can feel free to skip it.
</blockquote>
This will walk you through setting up remote desktop access to your Ubuntu Server using x11vnc and TigerVNC. I like using '''virtual machine manager''' GUI to install virtual machines on the main server. Since we keep going to the GUI to install virtual machines &amp; use '''virtual machine manager''' via the GUI, we’re stuck sitting in front of the server, which sucks if it’s in a closet or garage. Here’s how you can connect to it to view what is on its screen from another computer.

<span id="installing-x11vnc-on-ubuntu-server"></span>
==== 6.2 Installing x11vnc on Ubuntu Server ====

To install x11vnc, run the following command in your terminal:

<pre>sudo apt update &amp;&amp; sudo apt install x11vnc</pre>
This will install the x11vnc package and its dependencies on your server.

<span id="set-a-password-for-vnc-authentication"></span>
==== 6.3 Set a Password for VNC Authentication ====

x11vnc uses a password for authentication, and you can set this password as follows:

<pre>x11vnc -storepasswd</pre>
You will be prompted to enter a password. This password will be saved in the default location <code>~/.vnc/passwd</code>.

<span id="set-x11vnc-to-listen-on-all-interfaces-on-port-5920"></span>
==== 6.4 Set x11vnc to Listen on All Interfaces on Port 5920 ====

Open a terminal and run the following:

<pre>x11vnc -rfbport 5920 -usepw -auth ~/.Xauthority -display :0 -forever -norc -noxdamage -shared</pre>
Here is why this helps clients like Remmina connect:

'''1. <code>-rfbport 5920</code>'''

This sets the '''port''' on which the VNC server will listen for connections. VNC defaults to port <code>5900</code>, but I like to use a non-standard one because I am strange.

'''2. <code>-usepw</code>'''

This option enables '''password authentication''' for VNC clients. It requires you to set a password using <code>x11vnc -storepasswd</code> beforehand. Use this to set a VNC password.

* Password authentication is standard for VNC clients like Remmina. Without this, some clients might reject the connection for security reasons. And it’s just good practice.

'''3. <code>-auth ~/.Xauthority</code>'''

The <code>-auth</code> option tells <code>x11vnc</code> which '''authentication file''' to use to access your X session. The file path <code>/run/user/$(id -u)/gdm/Xauthority</code> refers to authorization for the user running the current display session managed by GDM (your display manager). If you’re using LightDM or another manager, the path might differ.

* '''Why it helps''': Instead of relying on <code>-auth guess</code> (which might not always find the right file), specifying the correct <code>Xauthority</code> file guarantees that <code>x11vnc</code> can properly access the graphical session. If <code>x11vnc</code> can’t authenticate the display, no client can connect.

'''4. <code>-display :0</code>''' This option specifies which '''X display''' to serve via VNC. The display <code>:0</code> is typically the primary display for your desktop session (the one you see on your monitor). It makes sure <code>x11vnc</code> is connecting to the right graphics session. If it were set to the wrong display, you’d either get a black screen or your client wouldn’t connect at all.

'''5. <code>-forever</code> ''' Normally, <code>x11vnc</code> stops running after the client disconnects. The <code>-forever</code> flag keeps it running indefinitely. If you disconnect &amp; reconnect it would suck to have to log back in each time. Without this, <code>x11vnc</code> would stop after Remmina disconnects, and you’d have to restart it manually for every new connection. I like stopping <code>x11vnc</code> once I am done manually.

'''6. <code>-norc</code> ''' This option tells <code>x11vnc</code> '''not to load a configuration file''' (which might contain unwanted settings), we are only using the settings in this command line.

'''7. <code>-noxdamage</code>''' The <code>Xdamage</code> extension tracks changes to the screen, but sometimes it can cause display corruption or update issues in VNC clients. The <code>-noxdamage</code> flag disables this extension to avoid those problems. Some VNC clients who shall not be named sometimes fk up refreshing the screen when properly when <code>Xdamage</code> is enabled. Disabling it keeps artifacts/stuck screen issues.

'''8. <code>-shared</code>''' This option allows '''multiple clients to connect simultaneously''' to the VNC server. If this option isn’t set, only one client can connect at a time, and additional connection attempts (such as from Remmina) would fail. Enabling <code>-shared</code> makes sure that you can connect with multiple devices or clients without being disconnected when another connects.

<span id="installing-tigervnc-viewer-on-the-client"></span>
==== 6.5 Installing TigerVNC Viewer on the Client ====

To connect to the VNC server from a client machine, you need a VNC viewer. The following steps will install TigerVNC Viewer (also known as <code>vncviewer</code>) on the client (your GNU/Linux computer you are reading this on):

<ol style="list-style-type: decimal;">
<li><p>Update the package list and install TigerVNC Viewer:</p>
<pre>sudo apt update &amp;&amp; sudo apt install tigervnc-viewer -y</pre></li>
<li><p>Once installed, you can use <code>vncviewer</code> to connect to the server.</p></li>
<li><p>If you use Windows or a Mac, you’re on your own, my friend. Find a VNC client that doesn’t suck.</p></li></ol>

<span id="connecting-to-the-vnc-server"></span>
==== 6.6 Connecting to the VNC Server ====

Now that everything is set up, you can connect to your server.

<ol style="list-style-type: decimal;">
<li><p>On your local machine, use the following command:</p>
<pre>vncviewer 192.168.5.2:5920 -SecurityTypes VncAuth</pre></li>
<li><p>''Note'': Replace <code>192.168.5.2</code> with your server’s actual IP address. In our case, we can also use the domain <code>happycloud.home.arpa</code> since we set up a static mapping earlier for our server in '''pfSense'''.</p></li>
<li><p>When prompted, enter the VNC password you set earlier.</p></li></ol>

'''You should now have a remote desktop connection to your Ubuntu Server. Remember to start <code>x11vnc</code> after you have logged in &amp; typed <code>startx</code> to start Openbox so it works.'''

<span id="step-7-using-openbox"></span>
== Step 7: Using Openbox ==

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:lu55028jxckj_tmp_a05cc5c9.png
</gallery>

Once you’ve installed Openbox and typed <code>startx</code>, Openbox starts:

# Right-click on the desktop to open the application menu.
# Navigate to '''System &gt; Virtual Machine Manager'''; This is what we are going to be using to create virtual machines.

At this point, we have our dependencies set up for virtual machine management, and bridge networking configured so that our virtual machines can go online. We’re ready to set up our first virtual machine!

<span id="creating-a-virtual-machine"></span>
= Creating a Virtual Machine =

Our first virtual machine will be for '''mailcow''' and '''bitwarden'''. These provide the following:

'''Mailcow''': - Self managed email server for sending &amp; receiving mail - Integrated spam management &amp; web interface - Calendar &amp; contacts syncing with mobile devices - A lovely, ''“just works”'' mashup of - Postfix for sending mail - Dovecot for receiving mail - rspamd for killing spam - SoGo for webmail/calendar/contacts

'''Bitwarden''': - Password management across devices, browsers, phones, computers, etc. - Alerts when your passwords have been found in a breach

<blockquote>'''Note:''' These instructions will carry over into many other virtual machine installs we will be doing. I will ask you to refer back to this section. Often, the only thing you will be doing is changing the RAM amount and CPU cores allotted to the VM, and the IP address you choose as you install.
</blockquote>
<span id="options-for-virtual-machine-creation"></span>
=== Options for Virtual Machine Creation ===

When you start creating a new virtual machine, you’ll see several options. We’re going to use “local install media” because we’re working with the ISO image of the Ubuntu server we downloaded. But before getting into that, let me explain the “import existing disk image” option, which is pretty cool :)

<span id="import-existing-disk-image"></span>
==== Import Existing Disk Image ====

Imagine you’ve got a bunch of old laptops lying around, each running different servers. Maybe you’ve got a Dell Latitude D620 from 2006 or a [https://www.youtube.com/watch?v=X86F1j5gCQs&pp=ygUcZnJlZXBieCByb3NzbWFubmdyb3VwIGxlbm92bw%3D%3D piece of junk Lenovo with a dying northbridge] running your entire business phone system; not that I ever did that.

But if you did, you could use a tool like <code>ddrescue</code> to make a disk image of each server. Then, you can import them into your virtual machine setup and keep them running without separate installations. It’s a useful method of consolidating everything onto one machine until you have time to set things up properly!

<span id="local-install-media"></span>
==== Local Install Media ====

This option expects us to choose a disk image (whether for a CD-ROM or a USB stick) that we will use to make a fresh installation onto our computer. This option is for when we want to create our own virtual machine from scratch, and is what we are going to be using.

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:lu55028jxckj_tmp_a05cc5c9.png
</gallery>

<span id="step-1-setting-up-virtual-machine-manager-virsh"></span>
== Step 1: Setting up Virtual Machine Manager (virsh) ==

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:lu55028jxckj_tmp_b51c10bd.png
</gallery>

<span id="create-new-virtual-machine"></span>
==== 1.0 Create new virtual machine ====

In '''Virtual Machine Manager''', click “Create a new virtual machine” (usually the first icon on the toolbar or select '''File &gt; New Virtual Machine''' from the menu).

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:lu55028jxckj_tmp_151745e3.png
File:lu55028jxckj_tmp_9a251511.png
File:lu55028jxckj_tmp_8bf9cc92.png
</gallery>

<span id="choose-installation-media"></span>
==== 1.1 Choose Installation Media ====

* Select '''“Local install media (ISO image or CDROM)”''' and click '''Forward'''.
* Click '''Browse''' to select your Ubuntu Server ISO.
* Choose the ISO file you prepared earlier (e.g., <code>/var/lib/libvirt/images/ubuntu-server.iso</code>) and click '''Forward'''.

<span id="choose-operating-system-version"></span>
==== 1.2 Choose Operating System Version ====

* '''Virtual Machine Manager''' may automatically detect the OS. If not, search for <code>ubuntu</code> and choose what is closest to your version. When in total doubt, <code>linux generic 2022</code> works. Click '''Forward'''.

<span id="configure-memory-and-cpu"></span>
==== 1.3 Configure Memory and CPU ====

* Allocate the resources for your VM:
** '''Set RAM:''' (e.g., 4096 MB).
** '''Set vCPUs:''' (e.g., 2 CPUs max for what we are doing here).
* Click '''Forward'''.

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:lu55028jxckj_tmp_52d81284.png
</gallery>

<span id="configure-storage"></span>
==== 1.4 Configure Storage ====

* Select “Create a disk image for the virtual machine”.
* Allocate an initial disk size that is whatever you think the maximum amount of storage you will need for email, contacts, and calendar is (e.g., 25 GB). You’ll be able to resize this disk later, so make sure it’s large enough for your initial installation but leave room for growth.
* Make sure the disk image format is '''QCOW2'''. This format supports resizing, and other cool features.
* Click '''Forward'''.

<blockquote>'''NOTE:''' QCOW2 format has a lot of useful features. It supports snapshots, which we aren’t using for our virtual machine backups, but it’s nice to have if you choose to use that. More importantly, qcow2 supports “sparse file allocation.” aka, it only uses physical disk space as it needs. Just because you say a virtual machine has access to 300 gigabytes, doesn’t mean it creates an image that actually takes up 300 gigabytes.
</blockquote>
<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:lu55028jxckj_tmp_a84796b3.png
</gallery>

<span id="set-up-networking-with-the-bridge-interface"></span>
==== 1.5 Set Up Networking with the Bridge Interface ====

* Choose '''“Bridge device”''' under '''“Network Selection”'''.
* In the Device Name field, type '''br0''' (or whatever name you have given your bridge interface).
* This will allow the VM to grab a static IP from the same network as your host machine, making sure it acts like an independent hardware device.
* Click '''“Finish”'''.

<blockquote>'''NOTE:''' Choosing “bridge device” allows the virtual machine to appear like a unique hardware device on your network. That’s the idea, for each of our virtual machines to seem like Pinnocchio; a real machine :)
</blockquote>
<div class="figure">

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:lu55028jxckj_tmp_bcbe1ce7.png
</gallery>

</div>
<span id="finish-customize-before-installing"></span>
==== 1.6 Finish &amp; Customize Before Installing ====

* Name your virtual machine (e.g., '''“mailserver”'''), whatever you think makes sense for a contacts/calendar/mail machine.
* Click '''“Finish”'''.

<div class="figure">

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:lu55028jxckj_tmp_f5a5b1fa.png
</gallery>

</div>
<span id="step-2-install-ubuntu-server-as-a-virtual-machine"></span>
== Step 2: Install Ubuntu Server as a Virtual Machine ==

<blockquote>'''Note:''' I will be blazing through the installing of Ubuntu here, since we already installed Ubuntu server once onto this physical server.
</blockquote>
'''Keep in mind the following:'''

We are NOT using LUKS encryption here. There is no need since the image is going to be stored on an encrypted partition.

We are NOT using RAID – this is a disk image that is being stored on a RAID array, so we are not doing that.

We are configuring networking the same as we did before, but we will be using a '''''different''''' IP address!

<blockquote>'''NOTE:''' Setting a unique static IP address to each virtual machine is necessary. It’s like having 5 businesses in the same building and expecting the postman to deliver laptops to the right address.. that’s never a problem that plagued me in new york city.

If something else is using that IP address, you cannot use it again. You don’t want your packets ending up in Berwick Maine.
</blockquote>
<span id="start-the-installation-process-in-the-virtual-machine"></span>
==== 2.1 Start the installation process in the virtual machine ====

Choose your language and select '''“Try or install Ubuntu Server”'''.

Follow the installation prompts.

<div class="figure">

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:lu55028jxckj_tmp_a561e59a.png
</gallery>

</div>
<span id="configure-static-ip-address"></span>
==== 2.2 Configure Static IP Address ====

* When you reach the Network configuration screen, select the network interface that corresponds to your network interface.
* Choose the option '''“Configure network manually”'''.
* Enter the following details:
** IP Address: '''192.168.5.3'''
** Subnet: '''192.168.5.0/24'''
** Gateway: '''192.168.5.1'''
** Nameserver: '''192.168.5.1'''
* Make sure you enter all the details correctly to provide the virtual machine has the correct static IP configuration.

<div class="figure">

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:lu55028jxckj_tmp_cd2383aa.png
</gallery>

</div>
<div class="figure">

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:lu55028jxckj_tmp_1d3ec660.png
</gallery>

</div>
<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:lu55028jxckj_tmp_e896bdf7.png
File:lu55028jxckj_tmp_b1313746.png
File:lu55028jxckj_tmp_664e174.png
File:lu55028jxckj_tmp_9a7daf09.png
File:lu55028jxckj_tmp_cd86d6ef.png
File:lu55028jxckj_tmp_9b21c766.png
</gallery>

<span id="partition-the-virtual-drive"></span>
==== 2.3 Partition the virtual “drive” ====

* When you reach the Filesystem setup section, select '''“Use an entire disk”''' and then choose the disk you want to install Ubuntu Server on.
* Choose the option '''“Set up this disk as an LVM group”'''.
* '''Important:''' At this stage, edit the partition sizes as Ubuntu’s installer usually allocates 2 GB for boot which is ridiculous and even worse it only uses half the available space for your LVM &amp; root. The Ubuntu auto partitioner is horrible.
* Reduce the boot partition to 512 MB.
* Delete the old LVM &amp; root partition.
* Create a new LVM taking up the entire disk.
* Create a logical volume for the root filesystem, using all available space.
* '''Do not encrypt the volume''' (it’s unnecessary since the host drive is already encrypted, and it is not my intention for you to have these VMs running on other people’s servers).

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:lu55028jxckj_tmp_b4178b46.png
File:lu55028jxckj_tmp_3d6c5298.png
File:lu55028jxckj_tmp_751040c0.png
</gallery>

<span id="finalize-installation-do-not-install-docker"></span>
==== 2.4 Finalize installation &amp; do not install docker ====

* Set up your username and password.
* '''Choose to install OpenSSH server.'''

<blockquote>'''NOTE:''' ''DO NOT CHOOSE TO INSTALL PACKAGES THROUGH THE PROMPTS AFTER THIS. THEY INSTALL VIA SNAP. DOCKER INSTALLED VIA SNAP IS CANCER. USING THE SNAP VERSION OF DOCKER WILL PROVIDE YOU WITH MANY AGGRAVATING HEADACHES. DON’T DO IT. IGNORE ME NOW? SUFFER LATER!''
</blockquote>
* After configuring the partition sizes, proceed with the installation process as usual, following the prompts to set up any additional software you want to install.
* Once the installation is complete, the system will automatically apply your network and partitioning settings.
* When prompted, remove the installation media (ISO) from the virtual machine settings.
* Restart the virtual machine.

<span id="step-3-post-installation-tasks"></span>
== Step 3: Post-Installation Tasks ==

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:lu55028jxdhn_tmp_ce86cb27.png
File:lu55028jxdhn_tmp_b291175e.png
File:lu55028jxdhn_tmp_b1c36016.png
</gallery>

<span id="remove-the-cdrom"></span>
==== 3.1 Remove the CDROM ====

* Go to '''View —&gt; details''' in virtual machine manager
* Go to '''SATA CDROM''' on the left side.
* Confirm that the '''source path''' is the Ubuntu ISO we downloaded for installing Ubuntu server on this virtual machine
* Click '''Remove''' in the lower right corner.
* UNCHECK '''Delete associated storage files''' – we will use this image again later!
* Click delete.
* You may have to turn off the VM to do this.

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:lu55028jxdhn_tmp_fb68028c.png
File:lu55028jxdhn_tmp_5504653d.png
File:lu55028jxdhn_tmp_22a2f148.png
File:lu55028jxdhn_tmp_75aac442.png
</gallery>

<span id="set-up-static-ip-mapping-in-pfsense-1"></span>
==== 3.2 Set Up Static IP Mapping in pfSense: ====

* Log into your '''pfSense''' router.
* Go to '''Status &gt; Diagnostics &gt; ARP Table'''.
* Find the MAC address associated with your server’s IP (e.g., '''192.168.5.3'''), copy it.
* Go to '''Services &gt; DHCP Server'''.
* Scroll to the bottom and click '''“Add static mapping”'''.
* Enter the MAC address and IP address of your server.
* Give it a descriptive name (e.g., “'''mailserver static IP'''”).
* Set the hostname to <code>mailserver</code>
* Save and apply changes.

''Note:'' This makes sure that this IP address is reserved for this computer to connect to, so that no other device can take it (unless they are spoofing MAC addresses, but if someone does, that’s a different story).

<span id="set-up-this-virtual-machine-to-start-at-boot"></span>
==== 3.3 Set up this virtual machine to start at boot: ====

<pre>virsh autostart mailserver</pre>
* Check that this is set up properly by typing <code>virsh dominfo mailserver</code> and seeing if the autostart line is set to enable.
* If you don’t do this, you will realize once it is too late &amp; you’ve left your house after you have rebooted your server (for whatever reason) that none of your services are working. This will suck.
* This command makes it so that the virtual machine starts each time we boot the computer.

<span id="calendar-contacts-using-sogo-within-mailcow"></span>
= Calendar &amp; Contacts using SoGo within Mailcow =

'''No more saving your contacts &amp; calendar to your Gmail account or iCloud – keep it all on your OWN server!'''

This guide will walk you through the process of installing and configuring '''mailcow''' on Ubuntu Server. Mailcow is an excellent solution for managing email, contacts, and calendars. It simplifies the setup of multiple mail-related services like <code>dovecot</code>, <code>rspamd</code>, <code>SpamAssassin</code>, <code>postfix</code>, <code>SoGo web interface</code>, <code>CalDAV</code>, making it easier ''(I will never use the word ''“easy”'' to describe self-managed email)'' to maintain a secure, working mail server with calendar &amp; contacts sync. Mailcow’s ease of use and strong community support make it perfect for self-hosting these services.

You will come to appreciate mailcow’s simplicity when we set up postfix manually for FreePBX &amp; ZFS filesystem alerts in later sections.

<span id="prerequisites"></span>
== Prerequisites: ==

<span id="for-self-hosted-calendar-contacts"></span>
=== For self-hosted calendar &amp; contacts: ===

* ''[https://mailcow.email/ mailcow]'' on your server
* ''[https://www.davx5.com/ DAVx⁵]'' on your phone
* A calendar app that works with ''[https://www.davx5.com/ DAVx⁵]'' such as ''[https://f-droid.org/en/packages/org.fossify.calendar/ Fossify Calendar]''

<span id="for-self-hosted-email"></span>
=== For self-hosted email: ===

* A domain name pointed to your server’s IP address that allows you to add TXT records, A records, etc.
* An SMTP relay provider such as ''[https://postmarkapp.com/blog/smtp-relay-services postmark]''
* More patience than ''[https://www.youtube.com/watch?v=DwQJT6Y7CyY&t=813s Rachel Cox waiting to leave Wind Cave]'' or John McCain ''[https://postmarkapp.com/blog/smtp-relay-services waiting to be rescued from prison in Vietnam]''

'''These instructions are going to serve as a base for each of our installations of a virtual machine that uses Ubuntu Server. I will ask you to refer back to these later.'''

<span id="step-1-prepare-ubuntu-server"></span>
== Step 1: Prepare Ubuntu Server ==

You can either work through virtual machine manager since virtual machine manager provides you a console view of your virtual machine, or <code>ssh</code> in from another computer.

<span id="update-and-upgrade-your-system"></span>
==== 1.1 Update and upgrade your system ====

<pre>sudo apt update &amp;&amp; sudo apt upgrade -y
sudo apt install curl git wget -y</pre>
<span id="check-for-other-docker-installations"></span>
==== 1.2 Check for other Docker installations: ====

Run <code>docker --version</code> and see what is installed. Nothing should be installed yet since this is a fresh system. If something is installed, remove it.

<pre># Just in case you accidentally installed snap version of docker:

sudo snap remove docker

For other versions of docker: 

sudo apt remove docker docker-engine docker.io containerd runc</pre>
<span id="install-docker-using-official-docker-script"></span>
==== 1.3 Install Docker using official Docker script: ====

<pre>curl -fsSL https://get.docker.com -o get-docker.sh
sudo sh get-docker.sh</pre>
<blockquote>
'''Note:''' It’s very important to use the official Docker installation and not the Snap version. The Snap version can cause issues due to its sandboxed nature, making it a mess for mailcow’s requirements. Docker snap makes me sad, and it’ll make you sad too if you try to make things work with it.
</blockquote>
<blockquote>
'''Editor's Note:''' Louis uses the ''[https://docs.docker.com/engine/install/ubuntu/#install-using-the-convenience-script convenience script]'' provided by Docker here. This method is <q>Only recommended for testing and development environments.</q> and may not update your dependencies correctly. For installation methods meant for a production environment see [https://docs.docker.com/engine/install/ubuntu/#installation-methods the official Docker manual].
</blockquote>
<span id="install-docker-compose"></span>
==== 1.4 Install Docker Compose: ====

Ubuntu’s <code>docker-compose-plugin</code> is safe to use, it is not snap cancer.

<pre>sudo apt install docker-compose-plugin -y
sudo systemctl enable --now docker</pre>
<span id="verify-the-install"></span>
==== 1.5 Verify the install ====

Run <code>docker compose version</code> and make sure the version is 2.0 or higher. Run <code>docker --version</code> and make sure version is 24.0.0 or higher

<span id="set-proper-permissions"></span>
==== 1.6 Set proper permissions: ====

Docker needs to be run as root for some operations, but you can add your user to the docker group to avoid using <code>sudo</code> all the time. To be clear, mailcow’s own [https://docs.mailcow.email/getstarted/install/#check-selinux-specifics documentation] and [https://community.mailcow.email/d/59-mailcow-containers-running-as-root community] suggest starting with root or <code>sudo</code>, and you should trust them more than me. To quote mailcow developers, ''“Controlling the Docker daemon as non-root user does not give you additional security. The unprivileged user will spawn the containers as root likewise. The behaviour of the stack is identical.”'' Run this command to add your user:

<pre>sudo usermod -aG docker $USER</pre>
Log out and log back in, or run:
<code>newgrp docker</code>

<span id="step-2-install-mailcow"></span>

== Step 2: Install mailcow ==

<span id="clone-the-mailcow-repository"></span>
==== 2.1 Clone the mailcow repository ====

<pre>cd /opt
sudo git clone https://github.com/mailcow/mailcow-dockerized
cd mailcow-dockerized</pre>
<span id="set-the-correct-permissions"></span>
==== 2.2 Set the correct permissions ====

Run <code>umask 0022</code>

<span id="generate-the-configuration-file"></span>
==== 2.3 Generate the configuration file ====

Run <code>sudo ./generate_config.sh</code>

When prompted, enter your Fully Qualified Domain Name (FQDN), such as <code>mail.yourdomain.com</code>.

<span id="start-mailcow-services"></span>
==== 2.4 Start mailcow services ====

<pre>sudo docker compose pull
sudo docker compose up -d</pre>
The <code>-d</code> option runs mailcow in detached mode so it continues running in the background.

<span id="verify-mailcow-install"></span>
==== 2.5 Verify the installation ====

Run
<pre>sudo docker ps</pre>
You should see a list of containers associated with mailcow running.

<span id="step-3-access-and-configure-mailcow"></span>
== Step 3: Access and Configure mailcow ==

Firstly, my screenshots from this guide were missing the top bar – my apologies. Refer to this below to see what was cut off, and hopefully where my mouse is clicking in both the rest of the screenshots &amp; the video will make sense:

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:lu55028jxdhn_tmp_c552f8ef.png
File:lu55028jxdhn_tmp_2c7243ee.png
</gallery>

<span id="access-the-web-interface"></span>
==== 3.1 Access the web interface ====

Open a browser and navigate to <code>https://mailserver.home.arpa</code>, replacing <code>your-mailcow-domain</code> with the hostname you set up when installing Ubuntu and making a static mapping. For example, with '''pfSense'''’s default domain ''home.arpa'', if the hostname is ''mailserver'', you can access it at <code>https://mailserver.home.arpa</code>. If unsure, use the IP address. If the IP is <code>192.168.5.3</code>, it would be <code>https://192.168.5.3/</code>. Ignore the warning about self-signed certificates since this is a new installation.

<span id="log-in-with-default-credentials"></span>
==== 3.2 Log in with default credentials ====

* '''Username:''' admin
* '''Password:''' moohoo

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:lu55028jxdhn_tmp_aa2eac37.png
File:lu55028jxdhn_tmp_40fcec0b.png
File:lu55028jxdhn_tmp_4c65a41d.png
</gallery>

<span id="change-the-administrative-password"></span>
==== 3.3 Change the administrative password ====

* Click on '''System → Configuration''' in the top menu.
* Click on '''Access → Administrators'''.
* Find the admin account and click the edit (pencil) icon.
* Enter a new, strong password.
* Click “Save changes”.

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:lu55028jxdhn_tmp_4255fdcb.png
File:lu55028jxdhn_tmp_a4d4f170.png
</gallery>

<span id="add-a-domain"></span>
==== 3.4 Add a domain ====

* Go to '''Email → Configuration''' on the top menu.
* In the '''Domains''' tab, click '''Add domain'''.
* Enter your domain name (e.g., yourdomain.com).
* Set any desired options (quota, aliases, etc.).
* Click '''Add domain''' (for example, <code>louishomeserver.chickenkiller.com</code>).

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:lu55028jxdhn_tmp_40fc5c1d.png
File:lu55028jxdhn_tmp_316dff73_2.png
File:lu55028jxdhn_tmp_f296577e.png
File:lu55028jxdhn_tmp_316dff73.png
</gallery>

<span id="add-an-email-account"></span>
==== 3.5 Add an email account ====

* Go to '''Email → Configuration → Mailboxes'''.
* In the '''Mailboxes''' tab, click '''Add mailbox'''.
** Enter the username (the part before @ in the email address).
** Choose the domain name (e.g., louis@yourdomain.com).
** Set a password for the mailbox.
** Make sure to check '''Grant direct login access to SOGo'''
*** This is what we use for webmail, calendar &amp; contacts
* Click '''Add ''' to add your mailbox.

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:lu55028jxdhn_tmp_f3c558c8.png
File:lu55028jxdhn_tmp_c65c7b62.png
File:lu55028jxdhn_tmp_49a98cdf.png
</gallery>

<span id="step-4-accessing-calendarcontacts-services"></span>
== Step 4: Accessing Calendar/Contacts Services ==

# Go to <code>https://mailserver.home.arpa</code>, or in this case, <code>https://192.168.5.3/</code> - this was the IP address &amp; hostname I suggested utilizing for this machine.
# Log in with your credentials.
# Click on “Apps” in the top right corner.
# Select '''SOGo''' to access webmail, calendar, and contacts.
# Alternatively, go directly to webmail by visiting <code>https://192.168.5.3/SoGo</code>. This can be faster, but the standard login section offers important spam control options so I suggest browsing around there first. '''SoGo''' is the web interface similar to <code>gmail.com</code>

<blockquote>'''Note:''' When logging in, make sure to use your full email address. This is necessary because mailcow supports multiple domains, so the full email address is required to identify the correct account.
</blockquote>
You can also set up your email client or mobile device using the configuration details provided in the mailcow interface.

You’ve now successfully set up mailcow on your Ubuntu Server. This is the base of a great self-hosted solution for email, calendars, and contacts.

Right now, we are not setting up email but focusing on calendar and contacts. For mobile access and syncing, we’re going to set up '''DAVx⁵''' on an Android device and configuring '''OpenVPN''' for secure remote access to your server. This will let you automatically sync calendar &amp; contacts from anywhere, for multiple calendars and multiple devices!

<span id="step-5-sync-android-with-mailcow-using-davx⁵"></span>
== Step 5: Sync Android with mailcow using DAVx⁵ ==

<span id="installing-davx⁵-on-an-android-phone"></span>
==== 5.1 Installing DAVx⁵ on an Android Phone ====

# Open the F-Droid store on your Android phone. If not installed, download it from https://f-droid.org/.
# In F-Droid, search for '''“DAVx⁵”'''.
# Locate DAVx⁵ in the results and tap on it.
# Tap the “Install” button to download and install .
# Once installed, open '''DAVx⁵'''.
# Grant all requested permissions when prompted. These typically include:
#* Access to contacts
#* Access to calendars
#* Access to storage
# You may see a donation request screen. While appreciated by developers, you can skip this for now. But they’re nice people, so think about giving them some money.

<span id="installing-fossifys-calendar-app-on-android-using-f-droid-store"></span>
==== 5.2. Installing Fossify’s Calendar App on Android Using F-Droid Store ====

# Open the F-Droid store on your Android phone.
# In the search bar, type '''“Calendar”''' and find the one made by '''fossify'''. You have to click the app sometimes to figure out who made it. It’s worth it. Their app is the only one that works properly.
# Press the “Install” button to download and install the app.

<blockquote>'''Note:''' Fossify Calendar is a fork of Simple Mobile Tools’ calendar app, maintained by developers who prioritize privacy and open-source principles. Simple mobile tools’ app was [https://www.reddit.com/r/SimpleMobileTools/comments/187w64x/simple_mobile_tools_bought_by_zipoapps/ bought by a cancerous spyware company]. IF YOU WERE USING OLD SIMPLE MOBILE TOOLS APPS – UNINSTALL THEM OR DO NOT ALLOW THEM TO AUTO UPDATE AGAIN, EVER.
</blockquote>
<span id="make-sure-android-phones-openvpn-connection-is-still-connected"></span>
==== 5.3. Make Sure Android Phone’s OpenVPN Connection is Still Connected ====

# Locate the OpenVPN Connect app on your Android phone.
# Open the app and check the connection status.
# If not connected, tap on the profile you created earlier (e.g., “Home VPN”).
# Tap the '''“Connect”''' button.
# Wait for the connection to establish. You should see a “Connected” status.

<blockquote>'''Important:''' Make sure you’re connected to your home network via OpenVPN before attempting to sync your contacts and calendar. If you’re not, it won’t find your server, since we haven’t forwarded any ports, and you are using local IP/hostnames to connect to it. Your router knows who <code>mailserver.home.arpa</code> is, your router knows who <code>192.168.5.3</code> is. To the outside world, this means nothing… and further, you’re not open to the outside world anyway.
</blockquote>
<blockquote>Think of it like the difference between saying “I want to find Sabrina Carpenter” to a hotel bellhop, vs. “I want to find my girlfriend.” Girlfriend only means something in reference to you. <code>Mailserver.home.arpa</code> only means something to you. The rest of the world has no idea who the fk that is.
</blockquote>
<div class="figure">

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:image-20241106170755552.png
</gallery>

</div>
<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:vlcsnap-2024-11-06-15h57m41s864.png
File:vlcsnap-2024-11-06-16h00m43s034.png
</gallery>

<div class="figure">

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:image-20241107032249683.png
</gallery>

</div>
<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:vlcsnap-2024-11-06-16h00m46s981.png
File:vlcsnap-2024-11-06-16h00m51s682.png
File:vlcsnap-2024-11-06-16h00m57s170.png
File:vlcsnap-2024-11-06-16h01m09s607.png
File:vlcsnap-2024-11-06-16h01m17s742.png
File:vlcsnap-2024-11-06-16h01m35s918.png
File:vlcsnap-2024-11-06-16h02m15s625.png
File:vlcsnap-2024-11-06-16h02m40s141.png
File:vlcsnap-2024-11-06-16h02m52s410.png
File:vlcsnap-2024-11-06-16h03m03s451.png
File:vlcsnap-2024-11-06-16h04m37s060.png
File:vlcsnap-2024-11-06-16h05m02s910.png
</gallery>

<span id="adding-mailcow-acct-to-your-phone-in-davx⁵"></span>
==== 5.4 Adding Mailcow acct to your phone in DAVx⁵ ====

<ol style="list-style-type: decimal;">
<li><p>Open the '''DAVx⁵''' app on your Android phone.</p></li>
<li><p>Tap on '''Add account''' to set up a new connection. ⊕</p></li>
<li><p>Choose “Login with URL and username”.</p></li>
<li><p>In the '''Base URL''' field, enter one of the following:</p>
<ul>
<li>Your server’s local IP address (e.g., <code>https://192.168.5.3</code>) or hostname (<code>https://mailserver.home.arpa</code>)</li></ul>

<blockquote><p>''Note:'' Use <code>https://</code> at the beginning of the URL for a secure connection. If it bitches at you, use <code>http://</code> - we’re connecting to this via OpenVPN which provides incredibly secure encryption anyway.</p></blockquote></li>
<li><p>Enter your login credentials:</p>
<ul>
<li>'''Username:''' Your full email address (e.g., <code>user@yourdomain.com</code>)</li>
<li>'''Password:''' Your mailcow account password</li></ul>
</li>
<li><p>Tap '''Login''' or '''Next''' to proceed.</p></li>
<li><p>If you see a certificate warning (due to a self-signed certificate), hit '''ACCEPT ''', this is your server. If you misfollowed something here so bad that you even have the ability to connect to someone else’s server right now, you amaze me more than the [https://expandingdan.substack.com/p/steely-dan-second-arrangement-erased-interview assistant engineer on the set of Gaucho].</p></li></ol>

<blockquote>'''NOTE:''' Self-signed certificates are common &amp; normal when setting up a home self managed server. They are not normal on the regular internet.

The entire point of a certificate is that a trusted certificate authority has deemed them to be them. When you go to amazon.com, someone authoritative is vouching that they are actually amazon, so some scammer can’t pretend to be amazon.com tomorrow.

For that authority to be able to vouch for amazon, they have to be able to '''ACCESS''' amazon.

We aren’t letting anyone access our server; and that’s the point. It’s only open via VPN - therefore, we can’t get a real certificate. You could open the port temporarily, and then close it right after you get the certificate, but that just feels dirty.

It’s fine to accept this warning for your OWN server; but don’t let this fly when you’re putting your credit card details or bank password into someone else’s website.
</blockquote>
<ol start="8" style="list-style-type: decimal;">
<li><p>When prompted for an account name, use your email address.</p></li>
<li><p>On the next screen, you’ll see options for syncing different data types:</p>
<ul>
<li>'''For Contacts:''' Enable “CardDAV” sync</li>
<li>'''For Calendar:''' Enable “CalDAV” sync</li>
<li>'''For Tasks (optional):''' Enable if you plan to use this feature, I don’t though.</li></ul>
</li>
<li><p>Tap “Create account” or “Finish” to complete the setup.</p></li></ol>

<div class="figure">

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:image-20241106163120676.png
</gallery>

</div>
<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:image-20241106163401141.png
</gallery>

<span id="adjusting-sync-settings"></span>
==== 5.5 Adjusting Sync Settings ====

After setting up your account, adjust the sync settings so you actually enjoy using this over Google/iCloud. The default sync interval is every 4 hours, which is horrible.

# In the DAVx⁵ app, find and tap on the account you just created.
# Look for sync settings, which will be in the settings, that you get to by clicking on the gear icon at the top of the application.
# Set up the sync intervals:
#* '''For server changes:''' Set to every 15 minutes (this is usually the minimum allowed interval)
#* '''For local changes:''' Set to immediate.
# Tap on each sync type (e.g., '''“Contacts”''' or '''“CardDAV”''').
# Look for sync interval settings within each category.
# Set server sync to 15 minutes and local changes to immediate for each.

'''Important Notes:'''

* The exact menu layouts and option names may vary slightly depending on your <code>DAVx⁵</code> version.
* Remember that for the 15-minute sync interval to work, make sure that <code>DAVx⁵</code> is exempted from battery optimization settings on your Android device. Android batteries are glued into the phone and most phones don’t let you limit charging to 80-90%, meaning the phone you’re using right now’s battery probably sucks and dies all the time anyway, might as well have up-to-date syncing on your contacts &amp; calendar.

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:image-20241106163429362.png
</gallery>

<div class="figure">

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:image-20241106163518587.png
</gallery>

</div>
<div class="figure">

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:image-20241106163545562.png
</gallery>

</div>
<div class="figure">

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:image-20241106163610419.png
</gallery>

</div>
<div class="figure">

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:image-20241106163632504.png
</gallery>

</div>
<div class="figure">

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:image-20241106163645629.png
</gallery>

</div>
<div class="figure">

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:image-20241106163709224.png
</gallery>

</div>
<div class="figure">

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:image-20241106163812905.png
</gallery>

</div>
<div class="figure">

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:image-20241106163835127.png
</gallery>

</div>
<div class="figure">

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:image-20241106163941177.png
</gallery>

</div>
<div class="figure">

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:image-20241106164009686.png
</gallery>

</div>
<div class="figure">

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:image-20241106164255822.png
</gallery>

</div>
<div class="figure">

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:image-20241106164453410.png
</gallery>

</div>
<span id="step-6-managing-contacts-with-mailcow-android"></span>
== Step 6: Managing Contacts with Mailcow &amp; Android ==

<span id="finding-your-new-mailcow-contacts-account-in-android"></span>
==== 6.1 Finding Your New Mailcow Contacts Account in Android ====

# Open the Contacts app on your Android phone.
# Tap on the menu icon (usually three lines or dots) to open settings.
# Go to '''Settings'''.
# Go to '''Accounts'''.
# Tap '''Add Account'''.
# Tap '''DAVx5 address book'''.
# Enter '''DAVx5''' app as it opens automatically, click checkbox.
# Once in the app, make sure your accounts are all selected &amp; checked.
# Return to the Android contacts app.
# Go to '''Settings —&gt; Accounts''' again.
# Do you see the green DAVx⁵ icon &amp; your account from Mailcow there? If so, great!
# Go back to '''Settings''' in the contacts app.
# Set up the '''default account for new contacts''' and the '''contacts to display''' so that your phone stores your contacts on your new Mailcow server to the account you created, and shows you contacts from your new server.
# Make sure this account is checked or toggled on to display its contacts.
# '''MAKE SURE YOU KNOW WHAT CONTACTS YOU ARE VIEWING &amp; WHERE THEY ARE BEING SAVED EARLY ON SO YOU DO NOT SCREW YOURSELF LATER! ON MY SETUP I EXPORT ALL OF MY CONTACTS TO A FILE, IMPORT THEM TO MAILCOW, AND AVOID USING THE PHONE FOR CONTACTS EVER. MORE PLACES YOU STORE CONTACTS = MORE CHANCES YOU SAVE TO THE WRONG PLACE &amp; SCREW YOURSELF LATER!'''

<div class="figure">

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:image-20241106164523457.png
</gallery>

</div>
<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:lu55028jxdhn_tmp_1f446aa8.png
File:lu55028jxdhn_tmp_26bc974a.png
File:lu55028jxdhn_tmp_e1f61714.png
File:lu55028jxdhn_tmp_f38b22b5.png
File:vlcsnap-2024-11-06-16h05m02s910.png
</gallery>

<div class="figure">

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:image-20241106164958898.png
</gallery>

</div>
<div class="figure">

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:image-20241106165036936.png
</gallery>

</div>
<div class="figure">

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:image-20241106165048430.png
</gallery>

</div>
<span id="adding-a-contact-in-mailcow-and-verifying-on-android"></span>
==== 6.2 Adding a Contact in Mailcow and Verifying on Android ====

# In the Mailcow SoGo web interface located at <code>https://192.168.5.3/SOGo/</code>, after logging in, find the option to add a new contact.
# Create a test contact with a unique name (e.g., “Test Mailcow Sync”).
# Save the new contact.
# On your Android phone, open the DAVx⁵ app and hit refresh.
# This will sync with Mailcow every 15 minutes, but if waiting 15 minutes.
# Yeah, I know, I know, Google &amp; iCloud have push… This is open source. We make sacrifices.
# When you add a contact on your PHONE, it will show up on the mailcow server in SOGo immediately. But the other way around takes 15 minutes.
# Open your Android Contacts app.
# Browse to the address book we just added. Or, do what I suggested above and stop using your device contacts list to begin with!
# Search for the unique name you gave the test contact.
# The contact should appear in your list, confirming that syncing from Mailcow to Android is working.
# Make sure this works both ways. Do not trust it until you test it. The worst thing in the world is losing a contact you thought you added. Ruby Lewis from Cirque Du Soleil could decide she wants to go out with you tomorrow—do you really want to lose her number because you messed up configuring <code>DAVx⁵</code>? I didn’t think so. It’s too easy to mess up this section not to double-check.

<blockquote>''Trivia: I quit Avatar Studios in 2008, after working there for a year as an intern, then junior technician in the tech room. I made $7.50/hr. Had I stayed there another 8 years, I wouldn’t have started a business, a YouTube channel, or made more than $15/hr; the other technician who had a master’s degree &amp; was 13 years my senior made $15. However, I would’ve gotten to say “hi” to Ruby Lewis in person. Would it have been worth it to not quit to meet my celebrity crush? Absolutely.''
</blockquote>
'''Important Notes:'''

* Make sure your OpenVPN connection is active if you’re not on your home network or this will not work. We intentionally set this server up to have no contact with the outside world with regards to contacts &amp; calendar syncing, so your phone must be connected to your home network via VPN for this to work.
* <code>DAVx⁵</code> typically syncs every 15 minutes, but you can force an immediate sync in the <code>DAVx⁵</code> app.
* If contacts don’t appear immediately, wait a few minutes or try forcing a sync in both <code>DAVx⁵</code> and your Contacts app.
* Remember to choose to save contacts in your Mailcow-linked address book to make sure they sync properly.

<div class="figure">

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:image-20241106170046456.png
</gallery>

</div>
<div class="figure">

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:image-20241106170349909.png
</gallery>

</div>
<div class="figure">

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:image-20241106170237454.png
</gallery>

</div>
<div class="figure">

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:image-20241106170546706.png
</gallery>

</div>
<span id="exporting-contacts-from-your-old-address-book"></span>
==== 6.3 Exporting Contacts from Your Old Address Book ====

# In your '''Android Contacts app''', go to '''Settings'''.
# Look for an option '''“Export”'''. Contacts app may be different from phone to phone, old version to new version, etc.
# Choose the account you want to export from (likely your old Google account or phone storage).
# Select '''Export to .vcf file'''.
# Choose a location to save the file, such as your phone’s Downloads folder.

<div class="figure">

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:image-20241106170604109.png
</gallery>

</div>
<div class="figure">

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:image-20241106170623962.png
</gallery>

</div>
<div class="figure">

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:image-20241106170635966.png
</gallery>

</div>
<div class="figure">

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:image-20241106170651317.png
</gallery>

</div>
<div class="figure">

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:image-20241106170701373.png
</gallery>

</div>
<div class="figure">

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:image-20241106171303341.png
</gallery>

</div>
<div class="figure">

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:image-20241106171535285.png
</gallery>

</div>
<div class="figure">

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:image-20241106171556372.png
</gallery>

</div>
<div class="figure">

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:image-20241106171747070.png
</gallery>

</div>
<div class="figure">

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:image-20241106172157386.png
</gallery>

</div>
<div class="figure">

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:image-20241106173758592.png
</gallery>

</div>
<div class="figure">

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:image-20241106174328393.png
</gallery>

</div>
<span id="importing-contacts-into-your-new-mailcow-address-book"></span>
==== 6.4 Importing Contacts into Your New Mailcow Address Book ====

# In the contacts app, click the three horizontal bars at the top you usually click before going to settings.
# Tap on your Mailcow account.
# Confirm that it only has the one contact that we added.
# Go back to the three-bar menu we were at before tapping the Mailcow account and tap settings.
# Find the option to “Import contacts”, usually called '''import.'''
# Select the <code>.vcf</code> file you exported earlier.
# Choose “DAVx⁵ personal address book” or your Mailcow-linked address book as the destination.
# Confirm the import. This process may take a few minutes depending on the number of contacts.
# Once it is done, customize your view by clicking “contacts to display” in your settings. Turn off EVERYTHING besides the DAVx⁵ Mailcow address book.
# Go back to the three-bar menu.
# Click onto your DAVx⁵ Mailcow address book. Do you see your contacts? It worked. :)

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:lu55028jxdhn_tmp_c7099f8b.png
</gallery>

<span id="verifying-contacts-in-mailcow-web-interface"></span>
==== 6.5 Verifying Contacts in Mailcow Web Interface ====

# On your computer, open a web browser and navigate to your Mailcow server’s address.
# Log in with your Mailcow credentials. Go to the webmail app, the SOGo thing.
# Look for the “Contacts” or “Address Book” section.
# You should see the contacts you just imported listed here. :D

<span id="step-7-setting-up-and-using-your-mailcow-calendar"></span>
== Step 7: Setting Up and Using Your Mailcow Calendar ==

<div class="figure">

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:image-20241106175124281.png
</gallery>

</div>
<div class="figure">

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:image-20241106175231132.png
</gallery>

</div>
<div class="figure">

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:image-20241106175249692.png
</gallery>

</div>
<div class="figure">

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:image-20241106175259734.png
</gallery>

</div>
<div class="figure">

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:image-20241106175311876.png
</gallery>

</div>
<div class="figure">

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:image-20241106175346849.png
</gallery>

</div>
<span id="configuring-fossify-calendar-app-with-davx⁵-synced-mailcow-calendar"></span>
==== 7.1. Configuring Fossify Calendar App with DAVx⁵-synced Mailcow Calendar ====

# Open the Fossify Calendar app on your Android phone.
# Tap the menu icon (usually three lines or dots) and select <code>Settings</code>.
# Check the box next to '''caldav sync'''.
# Tap '''Manage synced calendars'''.
# You should see a list of available calendars. Find the one associated with your Mailcow account and look for something with a familiar name to what you set up before.
# Make sure this calendar is checked or toggled on to display its events.
# If you don’t see your Mailcow calendar, go back to the <code>DAVx⁵</code> app, find your account, and make sure calendar sync is enabled.

<div class="figure">

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:image-20241106175409676.png
</gallery>

</div>
<div class="figure">

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:image-20241106175424361.png
</gallery>

</div>
<div class="figure">

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:image-20241106175544267.png
</gallery>

</div>
<div class="figure">

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:image-20241106175452608.png
</gallery>

</div>
<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:lu55028jxdhn_tmp_717bf114.png
</gallery>

<span id="adding-events-in-android-calendar-app-and-verifying-in-mailcow"></span>
==== 7.2 Adding Events in Android Calendar App and Verifying in Mailcow ====

# In the Fossify Calendar app, tap the “+” or '''Add event''' button.
# Enter event details:
#* '''Title''': Give it a unique name (e.g., “Test Android to Mailcow Sync”)
#* '''Date and time'''
#* Any other details you want to add

<blockquote>'''Important''': Make sure you select your Mailcow calendar as the destination calendar (not “Store locally only”). THIS IS VERY EASY TO MESS UP. PAY ATTENTION.
</blockquote>
<ol start="3" style="list-style-type: decimal;">
<li><p>Save the event.</p></li>
<li><p>Open a web browser and log into your Mailcow web interface.</p></li>
<li><p>Navigate to the calendar section.</p></li>
<li><p>You should see the event you just created appear in your Mailcow calendar. If it does not, you probably forgot to configure <code>DAVx⁵</code> properly so that it syncs on local changes immediately. Or you’re not on the VPN. Or you just messed up the configuration; do not pass go &amp; do not collect $200.</p></li></ol>

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:lu55028jxdhn_tmp_7b768f4d.png
File:lu55028jxdhn_tmp_58d82fba.png
File:lu55028jxdhn_tmp_d4b8aa27.png
</gallery>

<div class="figure">

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:image-20241106180513082.png
</gallery>

</div>
<div class="figure">

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:image-20241106180657084.png
</gallery>

</div>
<div class="figure">

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:image-20241106180744849.png
</gallery>

</div>
<span id="adding-events-in-mailcow-and-verifying-on-android"></span>
==== 7.3 Adding Events in Mailcow and Verifying on Android ====

# In your Mailcow web interface, navigate to the Calendar section.
# Find the option to add a new event (usually a “+” or '''New Event''' button).
# Create an event with a unique name (e.g., “Test Mailcow to Android Sync”).
# Set the date, time, and any other details.
# Save the event.
# On your Android phone, open the Fossify Calendar app.
# Swipe down or tap refresh.
# The new event should appear in your calendar view.
# '''''PSYCH!!!!'''''

<span id="refreshing-calendar-data"></span>
==== 7.4 Refreshing Calendar Data ====

<span id="refresh-button-in-calendar-app-is-not-real"></span>
===== Refresh button in Calendar app is not real =====

'''Refreshing directly in the Fossify Calendar app DOES NOT immediately show new events added on the server. For immediate updates:'''

# Open the DAVx⁵ app on your Android phone.
# Tap the '''Refresh''', and then the '''Synchronize Now''' button.
# Tap this to force an immediate sync with your Mailcow server.
# After the sync completes in DAVx⁵, open the Fossify Calendar app.
# Your calendar should now show the most up-to-date information.

You may wonder why this is, given that the calendar app literally has an option that says, “Refresh CalDAV Calendars.” that does not refresh your calendar. Welcome to the beautiful world of open-source software! :) I hope you’ll stay awhile. What we lack in functional UI, we make up for in not [https://docs.fcc.gov/public/attachments/FCC-24-40A2.pdf selling your data to bail bondsmen &amp; bounty hunters]. It’s kinda worth it…. kinda.

<span id="why-does-it-work-this-way"></span>
===== Why does it work this way? =====

When you tap '''“Refresh CalDAV Calendars”''', what you’re actually doing is asking the calendar app to check if anything has changed in CalDAV. You’re not telling CalDAV to contact your server to fetch new entries.

Here’s how it works:

# '''Mailcow server → CalDAV''' (Mailcow sends updates every 15 minutes)
# '''CalDAV → Calendar app''' (The calendar app pulls from CalDAV)

The calendar app will not immediately refresh unless you manually ask it to. And even when you do, it’s just checking DAVx⁵ for updates. It doesn’t ask DAVx⁵ to go and poll your Mailcow server.

* Remember that automatic syncs occur every 15 minutes by default.
* Always make sure you’re adding events to the correct calendar (your Mailcow calendar, not a local one).
* If you’re away from your home network, make sure your OpenVPN connection is active for the sync to work.
* If you experience any sync issues, check your internet connection and OpenVPN status, then try a manual refresh in DAVx⁵, NOT the calendar or contacts app first.

'''To force an immediate sync from the server at any time, you can tap refresh/sync now within the DAVx⁵ app or use a “Sync now” option if available.'''

'''THIS IS IMPORTANT: REFRESHING IN THE FOSSIFY CALENDAR APP WE INSTALL WILL NOT REFRESH INSTANTLY.'''

DAVx⁵ grabs data from our home server. Calendar &amp; contacts apps grab the data from DAVx⁵. When you tap '''“refresh”''' in your calendar app, what you’re ''actually'' doing is grabbing the latest data from DAVx⁵ on your phone. If DAVx⁵ does not have new data, it doesn’t matter if you just added a calendar event on your server &amp; you tap refresh furiously in the calendar app 50 times. The fossify calendar will still not see a new event on your server until '''DAVx⁵''' refreshes. Fossify does not have a way to trigger DAVx⁵ to refresh when you tap '''refresh''' in the fossify calendar app.

'''IF YOU WANT TO REFRESH TO SEE UPDATES IMMEDIATELY IN THE CALENDAR APP, YOU NEED TO HIT REFRESH/SYNC IN THE DAVx⁵ APP, ''THEN'' IN THE CALENDAR.'''

* I call this an ''“OPEN SOURCISM”'' - these are the byproducts of 20+ years of [https://www.reddit.com/r/immich/comments/1codh0p/comment/l5rfpu7/ people thinking it’s wrong for developers to get paid for their work]. It’s why Google &amp; Apple win; for all their flaws, they understand that developers want to be able to pay their rent &amp; feed their family in exchange for working 10 hours a day to produce software people use. There is only so much a small band of enthusiasts can do in their spare time, given that they need to make money to live indoors &amp; pay for food like the rest of us.
* If you want this to get better, show that you are willing to pay for software so people put time &amp; effort into fixing all of this.

By following these steps, you’ve now set up <code>DAVx⁵</code> to securely connect to your <code>mailcow</code> server and configured it to sync your data efficiently. As efficiently as it’ll let you; welcome to the world of self-managed open source servers! :D

<span id="self-managed-email-with-mailcow-postmark"></span>
= Self Managed Email with Mailcow &amp; Postmark =

Up to this point, we have only set up mailcow for contacts &amp; calendar syncing. '''This is as far as you should go.''' Self managed email is not for the faint of heart. If you are a beginner, do not pass go, do not collect $200, and skip on to the next section.

'''Choosing to do self managed email is like most of my relationship decisions:'''

# Just because you can doesn’t mean you should
# It’s messy, complicated, high maintenance.
# You’ll regret it later.

That being said, if you wish to continue…

<span id="why-do-i-need-smtp-relay"></span>
== Why do I need SMTP relay? ==

You need an SMTP relay server if you want people to actually see your email. No man is an island, and none of your mail is going to go anywhere without an SMTP relay. Gmail, etc., everyone will ''“lol''” at you if they see you sending email from your home email server.

As a society, we have chosen being spam-free over email sovereignty. You’re welcome to ''try'' running an email server on your residential internet account, but your mail is not going to get anywhere.

I’m not suggesting your email will end up in spam. It will be rejected by the server before its spam filter even sees it.

99% of the time that a major email server receives mail from a server on a residential internet connection, it’s from someone who got hacked &amp; is now unknowingly spamming half of the internet. We traded freedom to be rid of spam.

Whether or not you think this is fair is irrelevant; it’s how the world is. If you want your email to make it to most of your intended recipients, you need an SMTP relay.

SMTP relay sends your mail through postmark’s trusted server. Using postmark, icloud/gmail will let your mail through, rather than assume some schmuck running windows xp service pack 1 with his banking password post-it-noted to his monitor is part of a spam botnet.

Think of it like doing business in NYC. You are paying a troll toll for the ability to send email. But Postmark are nice people, so you’ll enjoy it. I hope they don’t cancel my services on account of me comparing them to [https://www.youtube.com/watch?v=qFVwQCFhKSE&list=PLkVbIsAWN2lvzWirQsz6haGEjJ2b2e6Ho&index=2 New York City government]. I’m sorry, postmark; that was uncalled for. :’(

<span id="step-1-setting-up-postmark-as-an-smtp-relay"></span>
== Step 1: Setting Up Postmark as an SMTP Relay ==

<span id="create-a-postmark-account"></span>
==== 1.1 Create a Postmark Account ====

* Go to: [https://postmarkapp.com postmarkapp.com]
* Sign up: Click on the '''Start free trial''' button at the top right-hand corner of the page.
** This is a paid service and you are going to pay, one way or another. If you don’t want to deal with forgetting you signed up for a trial, you can use [https://privacy.com privacy.com] to create a temporary credit card that is authorized for $50, then delete it the second you put it into Postmark. But if you choose to go the self-hosted email route, you will be paying; keep that in mind.
* Complete the registration: Enter the required details (email, password, etc.) and confirm your account through email verification.

'''Talk to Postmark; they need to know you are not a spammer.'''

* Postmark isn’t going to let you send email using their servers without taking them to dinner first. You need to get to know them &amp; they need to get to know you. They don’t let just ANYONE use their servers.
* This will take a day, or a few days, for them to verify that you are not a known spammer/scammer. This might require gentle nudging customer service if they do not get back to you quickly, but they usually do because Postmark is staffed by awesome people.
* They may ask for info about you. '''This is normal; no reputable SMTP relay wants to be responsible for helping deliver spam!'''

This may seem inconvenient, but it’s for the greater good of a spam free internet. If you don’t like that this is a thing, make sure to berate ''(verbally, of course)'' the next spammer you encounter. These people never refer to themselves by their proper name; they’re not ''“spammers,”'' they’re ''“email marketers.”''

'''If you check two of these three boxes, you are very likely a spammer, and have contributed to the amount of annoyance, aggravation, &amp; irritation that good people experience:'''

Are you responsible for sending me email that:

# utilizes templates
# includes in-line images
# has an “UNSUBSCRIBE” button

If you are, '''gargle my balls.'''

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:lu55028jxdmy_tmp_2dbf5b37.png
File:lu55028jxdmy_tmp_1b2ce54b.png
</gallery>

<span id="create-a-new-server"></span>
==== 1.2 Create a New Server ====

# Navigate to the '''Servers''' page:
#* After logging in, go to https://account.postmarkapp.com/servers or find the “Servers” tab in the top navigation bar.
# Create a new server:
#* Click on the '''Create Server''' button on the “Servers” page.
#* '''Name your server''': Enter a name for your pretty new SMTP relay server.
#* Click '''Save''' to create the server.

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:lu55028jxdmy_tmp_b5a39c43.png
</gallery>

<span id="configure-message-streams"></span>
==== 1.3 Configure Message Streams ====

# Navigate to the server you just set up by clicking on its name.
# Choose '''Default transaction stream''' from the three message streams it shows you.

<blockquote>'''Note:''' <code>Transactional</code> is for messages that are low volume but meant to be sent fast to an individual user, <code>broadcast</code> are for messages sent out to lots of users (aka spam) that are not time sensitive.
</blockquote>
<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:lu55028jxdmy_tmp_50cffcf2.png
</gallery>

<span id="get-smtp-relay-credentials"></span>
==== 1.4. Get SMTP Relay Credentials ====

# Navigate to the '''Setup Instructions''' page after clicking onto your message stream.
# If you forgot how to do this, you click '''Servers → Default Transactional stream → Setup Instructions'''.
# After configuring the outbound stream, go to the '''Setup Instructions''' page for the '''Transactional Outbound Stream'''.
# You will be overwhelmed with options under '''Pick the library or integration''' – no need to fear, we are picking '''SMTP'''.

'''SMTP details:'''

<ul>
<li><p>'''Server''': <code>smtp.postmarkapp.com</code></p></li>
<li><p>'''Ports''': 25, 2525, or 587. ''We will be using 587 with STARTTLS. You do not need to pick anything or configure anything here; this is just a page showing you your credentials you will put into Mailcow later. Save them securely. Pretend this is your bank password &amp; treat it accordingly.''</p></li>
<li><p>'''Authentication''': Postmark supports Plain Text, CRAM-MD5, or TLS.</p></li>
<li><p>'''Username''': This is your Postmark server token. It will look like a long string of characters (e.g., <code>1788dd83-9917-46e1-b90a-3b9a89c10bd7</code>).</p></li>
<li><p>'''Password''': The same value as the username (Postmark uses the server token as both the username and password).</p></li>
<li><blockquote><p>'''Note:''' As I go throughout this video, I will be using MY credentials as an example. THESE WILL NOT BE THE SAME AS YOURS. USE YOUR OWN CREDENTIALS.</p></blockquote></li></ul>

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:lu55028jxdmy_tmp_5a1ef5db.png
File:lu55028jxdmy_tmp_e265caf6.png
File:lu55028jxdmy_tmp_22cccc55.png
</gallery>

<span id="step-2-configuring-mailcow-to-use-postmark-as-smtp-relay"></span>
== Step 2: Configuring Mailcow to use Postmark as SMTP relay ==

<span id="access-mailcow-admin-interface"></span>
==== 2.1. Access Mailcow Admin Interface ====

# '''Login to Mailcow''':
# Navigate to your Mailcow instance by going to the admin interface URL (e.g., https://192.168.5.3/admin) or https://mailserver.home.arpa/admin.
# Use your administrator credentials to log in.

<span id="find-smtp-relay-section"></span>
==== 2.2. Find SMTP relay section ====

# From the main Mailcow admin dashboard, click '''System''' at the top and then click '''Configuration'''.
# Click onto the '''routing''' tab.
# Note the '''“add sender-dependent transport”''' section. This is where we will be placing our Postmark credentials.

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:lu55028jxdmy_tmp_e2f517bd.png
File:lu55028jxdmy_tmp_d0f1df1f.png
File:lu55028jxdmy_tmp_7f812343.png
File:lu55028jxdmy_tmp_259c85b9.png
File:lu55028jxdmy_tmp_85086323.png
</gallery>

<span id="enter-postmark-smtp-details"></span>
==== 2.3 Enter Postmark SMTP Details ====

# Use the credentials provided by Postmark in the prior step, which have a screenshot included.
#* '''SMTP Server''': Set the SMTP server to Postmark’s SMTP, which at the time of writing for me was <code>smtp.postmarkapp.com:587</code>.
#* '''Ports''': If Postmark is still using port 587 for TLS and offering it at the time of this writing, use port 587.
#* '''Username &amp; Password''': Enter your Postmark server token (the token provided by Postmark when you created your server). This token serves as both the username and password. This is what you see on the '''servers —&gt; default transactional stream —&gt; setup instructions —&gt; SMTP''' page under ''“Authenticate with a server token and specify stream with a header”''
#* '''Example''':
#** '''Username''': <code>1788dd83-9917-46e1-b90a-3b9a89c10bd7</code> (replace with your actual token).
#** '''Password''': Same as the username (server token).
# Click '''Add'''.

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:lu55028jxdmy_tmp_c2bfd8f9.png
File:lu55028jxdmy_tmp_146d196d.png
File:lu55028jxdmy_tmp_93c26e5c.png
</gallery>

<span id="step-3-adding-a-domain-name-mailbox-to-mailcow"></span>
== Step 3: Adding a Domain Name &amp; Mailbox to Mailcow ==

<span id="add-a-domain-1"></span>
==== 3.1. Add a Domain ====

# Go to '''Email → Configuration''' on the top menu.
# Go to the '''Domains''' tab.
# In the '''Domains''' tab, click '''Add domain'''
# Enter your domain name (in my case, stevesavers.com).
# Set any desired options (quota, aliases, etc.).
# Make sure DKIM key length is at least 2048.
# Click '''Add domain and restart SOGo'''.

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:lu55028jxdmy_tmp_74c2130f.png
File:lu55028jxdmy_tmp_d74edddb.png
File:lu55028jxdmy_tmp_1aa99468.png
File:lu55028jxdmy_tmp_30fff2e9.png
File:lu55028jxdmy_tmp_6eaf338a.png
</gallery>

<span id="set-postmark-as-the-relay"></span>
==== 3.2 Set Postmark as the Relay ====

'''IF YOU DO NOT DO THIS, NONE OF YOUR EMAIL WILL SEND!'''

* Click '''Edit''' on the domain name you just created.
* Now you will see a NEW option: '''sender-dependent transports'''.
* In the domain settings, find the option labeled '''sender-dependent transports''' and select the newly created Postmark relay (e.g., <code>smtp:postmarkapp.com</code>). Set this to the Postmark SMTP relay server you set up in the prior step. Sometimes this is already checked for you, but it is safe to '''''inspect what you expect''''' so you don’t get screwed!

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:lu55028jxdmy_tmp_e994614f.png
File:lu55028jxdmy_tmp_6e65b29c.png
File:lu55028jxdmy_tmp_10bee612.png
File:lu55028jxdmy_tmp_42594faf.png
File:lu55028jxdmy_tmp_aa150a6e.png
</gallery>

<span id="add-an-email-account-1"></span>
==== 3.3. Add an Email Account ====

* Go to'''Email → Configuration → Mailboxes'''.
* In the '''Mailboxes''' tab, click '''Add mailbox'''.
* Enter the username (the part before @ in the email address).
* Choose the domain name (e.g., <code>louis@yourdomain.com</code>).
* Set a password for the mailbox.
* Configure any additional options as you want.
* Click '''Add mailbox'''.

<span id="save-changes-and-apply"></span>
==== 3.4 Save Changes and Apply ====

* After choosing the <code>smtp.postmarkapp.com:587</code> SMTP relay, click '''Save Changes''' to apply the settings.

<span id="accessing-sogo-webmailcalendarcontacts"></span>
==== 3.5 Accessing SoGo Webmail/calendar/contacts ====

# Go to https://mailserver.home.arpa, or in this case, https://192.168.5.3/SoGo.
# Log in with your credentials.
# Click on '''Apps''' in the top right corner.

<blockquote>'''Note''': When logging in, make sure to use your full email address. This is necessary because Mailcow supports multiple domains, so the full email address is required to identify the correct account.
</blockquote>
You can also set up your email client or mobile device using the configuration details provided in the Mailcow interface.

<span id="step-4-setting-up-dns-records-in-your-domain-registrar"></span>
== Step 4: Setting up DNS Records in your domain registrar ==

<span id="introduction-to-domain-registrars"></span>
=== Introduction to domain registrars ===

<span id="what-is-a-domain-registrar"></span>
==== What is a domain registrar? ====

This is who you buy your website name from. If you don’t know what this is… for the love of god skip the self-hosted email section.

<span id="namecheap.com-as-an-example"></span>
==== Namecheap.com as an example ====

Namecheap is a cheap &amp; easy way to register a domain name. I will use them as an example. Their interface for DNS configuration is similar to 99% of the available providers out there.

If you have any trouble setting up these records, contact the support staff of your domain name provider who will happily provide you tech support commensurate with the fifteen dollars per year you pay them. No really, you’re on your own here… do you ''really'' want to do this??

I would love to show you how to do this on every provider, but at this time this manual is 605 pages, the video is 12+ hours, and I would like to return to my life. You will be able to find similar settings, menus, and fields in your DNS registrar if your provider isn’t horrible.

<span id="configuring-dns-records-in-namecheap"></span>
=== Configuring DNS records in Namecheap ===

<div class="figure">

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:lu55028jxdmy_tmp_2c3c94e.png
</gallery>

</div>
<div class="figure">

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:lu55028jxdmy_tmp_703091b1.png
</gallery>

</div>
<span id="find-the-dkim-thing-for-your-domain"></span>
==== 4.1. Find the DKIM thing for your domain ====

# Go to '''Email → Configuration''' on the top menu.
# Go to the '''Domains''' tab.
# In the '''Domains''' tab, click '''edit''' on the domain you created (in my case, stevesavers.com).
# Scroll down to the DKIM section. Keep this tab open for now; we will come back to it later.
# We’re not changing anything here, so there’s no need to save changes or make any changes. We just want that DKIM thing.

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:lu55028jxdmy_tmp_ffbba2cb.png
</gallery>

<span id="configure-dns-records-in-namecheap"></span>
==== 4.2 Configure DNS records in Namecheap ====

# Log into your Namecheap.com account.
# Go to Domain List and click '''Manage''' next to your domain.
# Navigate to the '''Advanced DNS''' tab.
# Here are the DNS records I added: you will fill them according to your specific setup. <gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:lu55028jxdmy_tmp_39fbaebb.png
</gallery>

<span id="cname-record"></span>
===== CNAME Record =====

* '''Host''': <code>pm-bounces</code> (Keep this exactly the same)
* '''Value''': <code>pm.mtasv.net.</code> (Keep this exactly the same)
* '''TTL''': Automatic (Keep this the same unless your DNS provider requires a different TTL setting)

This CNAME record is used by Postmark for handling email bounces. When an email bounces, it will be sent to <code>pm-bounces.[yourdomain]</code>, which forwards the bounce to Postmark’s servers. No changes are needed unless you are using a different bounce-handling service.

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:lu55028jxdmy_tmp_22a6cdb4.png
</gallery>

<span id="dmarc-record-txt"></span>
===== DMARC Record (TXT) =====

* '''Host''': <code>_dmarc</code> (Keep this exactly the same)
* '''Value''': <code>v=DMARC1; p=none; rua=mailto:dmarc@stevesavers.com</code> ''(Change only the email address after <code>rua=mailto:</code> to your own)''

Here’s what stays the same and what changes:

* <code>v=DMARC1</code>: (Keep this exactly the same)
* <code>p=none</code>: (Keep this exactly the same for monitoring; change to <code>p=quarantine</code> or <code>p=reject</code> once you’re ready to enforce DMARC)
* <code>rua=mailto:</code> [mailto:dmarc@stevesavers.com '''dmarc@stevesavers.com''']: Change <code>stevesavers.com</code> to your own domain and use an email where you want to receive DMARC reports.

This DMARC record helps protect your domain from email spoofing. For now, it’s in monitoring mode, so keep <code>p=none</code> if you want to monitor. If you’re ready to enforce policy, change <code>p=none</code> to <code>p=quarantine</code> or <code>p=reject</code>.

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:lu55028jxdmy_tmp_aa66e26f.png
File:lu55028jxdmy_tmp_257280e5.png
File:lu55028jxdmy_tmp_edea6316.png
File:lu55028jxdmy_tmp_65dc145a.png
File:lu55028jxdmy_tmp_7c22f73c.png
File:lu55028jxdmy_tmp_93ecca45.png
File:lu55028jxdmy_tmp_62fd886c.png
File:lu55028jxdmy_tmp_2a58b7ee.png
</gallery>

<span id="postmark-dkim-record-txt"></span>
===== Postmark DKIM Record (TXT) =====

This you are going to get by doing as follows:

# Go to postmark.com and log in
# Go to your domain interface, go to '''Sender Signatures''', click '''Add Domain or Signature''', then '''Add Sender Signature'''.
# Once you’re done it’ll present you with a DKIM record and a return path. I’ll show you what we’re doing with these below &amp; in the attached pictures:

<blockquote>'''Note:''' When adding your domain, choose to send from any email address on the domain, not just a single one.
</blockquote>
* '''Host''': <code>20241012215824pm._domainkey</code> (Postmark generates this value, so keep it exactly as provided by Postmark)
* '''Value''': <code>k=rsa; p=MIGfMA0GCSq...</code> (You will replace the long key string <code>p=</code> with the public key provided by Postmark)

<blockquote>'''IMPORTANT:''' The Host (<code>20241012215824pm._domainkey</code>) and <code>k=rsa</code> are specific to Postmark and should stay the same. You need to copy and paste this key exactly as Postmark provides it '''FROM POSTMARK, NOT FROM THIS GUIDE!'''
</blockquote>
<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:lu55028jxdmy_tmp_ab1378ba.png
File:lu55028jxdmy_tmp_ba775df9.png
File:lu55028jxdmy_tmp_3ba69113.png
File:lu55028jxdmy_tmp_d073948d.png
File:lu55028jxdmy_tmp_72c9d18.png
File:lu55028jxdmy_tmp_ccb1f143.png
File:lu55028jxdmy_tmp_d4f449eb.png
</gallery>

<span id="dkim-record-for-your-domain-txt"></span>
===== DKIM Record for Your Domain (TXT) =====

# Log into mailcow’s administration interface.
# Go to '''Email → Configuration''' on the top menu.
# Go to the '''Domains''' tab.
# In the '''Domains''' tab, click '''edit''' on the domain you created (in my case, stevesavers.com).
# Scroll down to the DKIM section.
# Insert the record as follows:
#* '''Host''': <code>dkim._domainkey</code> (Keep this exactly the same unless mailcow email provider tells you to use a different prefix)
#* '''Value''': <code>v=DKIM1; k=rsa; t=s; s=email; p=MIIBIjANB...</code> (Replace this with the figure)

<blockquote>The Host should be <code>dkim._domainkey</code> unless your email provider asks for a different format. For the Value, keep <code>v=DKIM1; k=rsa; t=s; s=email</code> exactly the same. The part you need to change is the long public key string after <code>p=</code>, which will be provided by your email provider or mail server (like Mailcow). Copy and paste it carefully.
</blockquote>
<span id="spf-record-txt"></span>
===== SPF Record (TXT) =====

* '''Host''': <code>@</code> (Keep this exactly the same)
* '''Value''': <code>v=spf1 mx a include:spf.mtasv.net ~all</code> (Enter this as it is: change the include value if using a different SMTP service than postmark or if [https://postmarkapp.com/glossary/sender-policy-framework postmark changes this in the future])

Here’s what stays the same and what you need to change:

* '''Host''': Always use <code>@</code> for your main domain.
* '''Value''':
** <code>v=spf1 mx a</code>: Keep this exactly the same; it tells servers to check your MX and A records.
* <code>include:spf.mtasv.net</code>: You will need to change this if you’re using a different mail service than Postmark. Replace <code>spf.mtasv.net</code> with the SPF record provided by your SMTP service (e.g., if using a different relay like SendGrid or Amazon SES, they will give you a different include value).
* <code>~all</code>: Keep this the same unless you want stricter enforcement. You can replace <code>~all</code> with <code>-all</code> for stricter failure rules.

<span id="mail-cname-record"></span>
===== Mail CNAME Record =====

* '''Host''': mail (Keep this exactly the same)
* '''Value''': <code>louishomeserver.chickenkiller.com.</code> (Change this to the domain or subdomain that hosts your mail server, '''this is what you set when you created a dynamic DNS domain at freedns!''')

<blockquote>The Host mail stays the same. What you will change is the value after <code>Value:</code>, which should point to the domain or subdomain that hosts your mail server. Replace <code>louishomeserver.chickenkiller.com</code> with your actual mail server’s domain or subdomain.
</blockquote>
<span id="email-client-configuration-cname-records"></span>
===== Email Client Configuration CNAME Records =====

* '''Host''': autoconfig (Keep this exactly the same)
* '''Value''': <code>mail.stevesavers.com.</code> (Change this to the domain of your mail server)
* '''Host''': autodiscover (Keep this exactly the same)
* '''Value''': <code>mail.stevesavers.com.</code> (Change this to the domain of your mail server)

<blockquote>Both Host fields (autoconfig and autodiscover) stay the same, as they are used for automatic email client configuration. You will change the Value to point to your mail server’s domain or subdomain (in this case, <code>mail.stevesavers.com</code>). Replace this with your own mail server domain.
</blockquote>
<span id="mx-record"></span>
===== MX Record =====

* '''Host''': @ (Keep this exactly the same)
* '''Value''': <code>mail.stevesavers.com.</code> (Change this to the domain of your mail server)
* '''TTL''': Automatic (Keep this the same unless your DNS provider requires a specific TTL)

The Host @ stays the same to apply to your root domain. What you need to change is the value after <code>Value:</code>, which should point to the domain that handles incoming mail for your domain. Replace <code>mail.stevesavers.com</code> with your own mail server domain.

These DNS records set up email services for your domain. For the third time, here’s what stays the same and what needs changing:

* '''SPF, DKIM, and DMARC''': Most parts of these records remain the same, but you’ll need to customize the DKIM public keys and the domain-specific parts (like email addresses for DMARC reports or SPF includes).
* '''MX and CNAME records''': The basic structure stays the same, but you’ll need to update the domain values to point to your own mail server.

By carefully adjusting the fields noted for customization, you can provide the DNS setup matches your unique mail and web infrastructure.

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:lu55028jxdmy_tmp_3e7d5187.png
File:lu55028jxdmy_tmp_841a3e85.png
File:lu55028jxdmy_tmp_6d09e55d.png
File:lu55028jxdmy_tmp_3abfd2ad.png
</gallery>

<span id="go-back-to-postmark-verify-your-dns-records."></span>
==== 4.3 Go back to Postmark &amp; verify your DNS records. ====

# Go to postmark.com and log in.
# Go to your domain interface, go to '''Sender Signatures'''.
# Click onto the ones you just created.
# Click '''VERIFY''' next to both '''DKIM''' and '''Return Path.'''
# If it doesn’t work yet, no big deal, DNS changes can take time to propagate.

<span id="step-5-pfsense-firewall-introduction"></span>
== Step 5: pfSense firewall introduction ==

So you have a basic idea on how to use '''pfSense''' as a basic router, but we haven’t dealt with '''''port forwarding''''' or messing with the firewall yet. Let’s get into that.

Before we move on to making the necessary firewall rules to allow us to receive email, let’s discuss aliases.

What makes firewall rules easy to manage are '''aliases.'''

<span id="lesson-1-aliases-in-pfsense"></span>
=== Lesson 1: Aliases in pfSense ===

<span id="what-are-aliases-in-pfsense"></span>
===== What are Aliases in pfSense? =====

Aliases in '''pfSense''' are placeholders that can represent:

* IP addresses
* Networks
* Ports
* URLs

For example, instead of having to make a separate NAT &amp; firewall rule to open port 993 for 8.8.8.8, 9.9.9.9, 10.10.10.10, etc., I can make ONE firewall rule and enter the “alias” I created into the field where I would usually put an IP. I’d create an alias for those three IPs.

The cool part about this is if I ever want to add or remove one of those IPs, I don’t have to change firewall rules or delete/add firewall rules. I just change my alias.

<span id="practical-example"></span>
===== Practical example: =====

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:lu55028jxdmy_tmp_942c4249.png
</gallery>

* If you’re using a service like Freshdesk (CRM system):
* Freshdesk needs to connect to your mail server
* You don’t want to give Freshdesk VPN access
* Freshdesk doesn’t have VPN access anyway
* Here’s how you can handle this situation:
* Add their IPs to your alias
* Only those IPs will see your mail server
* Everyone else gets blocked before even seeing the service
* Using aliases this way means:
* Your mail server is invisible to random internet traffic
* Only trusted IPs can even attempt connection
* Much more secure than opening ports to everyone

<blockquote>'''IMPORTANT:''' While port 25 needs to be open to the world for receiving email, other mail-related ports (587, 993, etc.) should only be open to trusted IPs or VPN users.Let’s say I am making firewall rules to allow Freshdesk customer service software to access my email system. Can you imagine making a firewall rule for EACH of these individually
</blockquote>
Can you imagine having to add all of those IPs as its own separate rule, or having to update them all each time freshdesk’s IPs changed? That would be a nightmare!

Aliases allow us to add all of these IP addresses to a single thing called ''“freshdesk IP addresses”'' – then, all we have to do is make a firewall rule with ''“freshdesk IP addresses”'' as the source or destination, rather than a bunch of rules for each individual IP.

<span id="benefits-of-using-aliases"></span>
===== Benefits of Using Aliases =====

# '''Simplification''': Instead of entering “192.168.5.3” into a firewall rule, if I make an alias, I can just enter “mailserver”, once I have set up a “mailserver” alias that directs to the mailserver.
# '''I can add to it!''' Let’s say I have 1 smart television in my house. I want to block it from going onto the internet to ''anything'' besides a single Netflix IP address, so I add a firewall rule to block it from going online to anything besides the Netflix IP address. Let’s say my family buys 3 more smart TVs… I don’t want to set up a new set of firewall rules each time. Aliases allow me to add '''''multiple IP addresses to a single alias!''''' Instead of having to make 5 new sets of rules, I can keep my existing firewall rules as they are, and simply add the new IP addresses to the alias.
# '''Maintainability''': When you need to update multiple firewall rules, you can just update the alias instead of each individual rule.
# '''Readability''': Aliases make firewall rules more understandable by using descriptive names instead of raw IP addresses or port numbers.

<span id="wtf-openvpn-was-set-up-so-we-dont-open-ports-why-are-we-talking-about-opening-ports"></span>
===== WTF? OpenVPN was set up so we DON’T open ports; why are we talking about opening ports? =====

If you are accessing your mailserver using OpenVPN (AS YOU SHOULD), this doesn’t matter. You will be opening port 25 to the world so you can receive email, but for the rest of the ports, these are ONLY NECESSARY IF YOU WANT CLIENTS WHO ARE NOT CONNECTING TO YOUR VPN TO BE ABLE TO LOG INTO AN EMAIL ACCOUNT AND READ THEIR MAIL AND SEND MAIL ON YOUR MAILSERVER!!!

Plus, the self-hosted phone system is going to require we allow some external IPs belonging to our SIP trunking provider (the thing that lets you receive &amp; send calls to other phones outside your house) to access our server anyway, so you might as well learn about aliases now.

<span id="how-to-set-up-aliases-in-pfsense"></span>
==== How to Set Up Aliases in pfSense ====

<span id="accessing-the-aliases-page"></span>
===== 5.1.1 Accessing the Aliases Page =====

# Log into the '''pfSense''' web interface.
# Navigate to '''Firewall &gt; Aliases'''.
# Click '''Add'''

<span id="creating-an-alias"></span>
===== 5.1.2 Creating an Alias =====

# In the Name field, enter a descriptive name for your alias (e.g., “WebServers” or “BlockedIPs”).
# Select the Type of alias you want to create:
#* Host: For single IP addresses
#* Network: For subnets
#* Port: For port numbers
#* URL: For lists of IPs or networks from a URL
# In the Description field, enter a brief explanation of the alias’s purpose. Here, I would enter <code>mailserver</code>.
# In the Content box, enter the values for your alias:
#* For IP aliases: Enter IP addresses, one per line, such as our mailserver at <code>192.168.5.2</code>.

<span id="using-aliases-in-firewall-rules"></span>
===== 5.1.3 Using Aliases in Firewall Rules =====

# Go to '''Firewall &gt; NAT'''.
# Add a new rule or edit an existing one.
# In the source or destination fields, you can now select your alias from the drop-down menu.
# For port fields, you can select port aliases.

Example rule using aliases:

* '''Action''': Pass
* '''Interface''': WAN
* '''Source''': Any
* '''Destination''': WebServers (alias)
* '''Destination Port''': WebPorts (alias)

This rule allows incoming traffic to the IP addresses defined in the <code>WebServers</code> alias on the ports defined in the <code>WebPorts</code> alias.

<span id="using-aliases-for-secure-access"></span>
=== Using Aliases for Secure Access ===

If you want external access to your mail server ''without'' requiring VPN, you’ll need to set up aliases for trusted IPs; or open your server to the entire world, which is a poor idea.

<span id="lesson-2-setting-up-pfsense-firewall-rules-for-a-mail-server"></span>
=== Lesson 2: Setting Up pfSense Firewall Rules for a Mail Server ===

<span id="understanding-nat-vs.-firewall-rules"></span>
===== Understanding NAT vs. Firewall Rules =====

Let’s understand the two types of rules you need to set up in '''pfSense''':

<span id="nat-network-address-translation"></span>
===== NAT (Network Address Translation) =====

NAT determines ''where'' traffic goes. Here’s why it matters:

* Your network has one public IP that the world sees
* But you might have 200+ computers internally
* When someone sends you an email, NAT tells the router ''“traffic on port 25 goes to the mail server, port 80 goes to the web server”'' etc.

Think of NAT like a restaurant host - they decide which table gets which customers.

<span id="firewall-rules"></span>
===== Firewall Rules =====

Firewall rules determine if traffic is ''allowed'' to pass. After NAT directs traffic to a computer, firewall rules decide if it gets through.

Think of firewall rules like the bouncer - they decide if you get in at all.

<span id="practical-application"></span>
===== Practical Application =====

'''NAT port forward''' is when the router sees an email coming in on port 25 to my spectrum internet address, and sends that email to our mail server on port 25.

Once NAT has sent that email to my mailserver on port 25, the '''firewall rule''' is what '''allows''' that traffic to access port 25 on our mailserver.

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:lu55028jxdmy_tmp_1f1b2c6a.png
File:lu55028jxdmy_tmp_31037c49.png
File:lu55028jxdmy_tmp_8d77cc05.png
File:lu55028jxdmy_tmp_54c8a9f1.png
File:lu55028jxdmy_tmp_17e7dded.png
File:lu55028jxdmy_tmp_cc025f84.png
</gallery>

<span id="setting-up-mail-server-port-forwarding-so-you-receive-emails"></span>
==== Setting Up Mail Server Port Forwarding so you Receive emails: ====

A “mail client” is a program you use to read &amp; send your email from the mail server (the mailcow machine we are setting up). Examples are k9 mail, Microsoft Outlook, Mozilla Thunderbird, etc., or just using the web interface.

If you are going to use the mail server while connected to the VPN, '''''THIS IS THE ONLY RULE YOU NEED TO ADD!''''' This is for '''receiving email.''' This port '''''must''''' be opened to the public.

<span id="create-nat-rule"></span>
===== Create NAT Rule =====

# Access '''pfSense''' at <code>https://192.168.5.1</code>
# Go to '''Firewall → NAT'''
# Under the '''Port Forward''' tab, click '''Add'''
# Configure the following:
#* '''Interface''': WAN (incoming traffic)
#* '''Protocol''': TCP
#* '''Source''': Any ''(you can’t predict which mail servers will email you)''
#* '''Destination''': WAN address
#* '''Destination Port Range''': 25
#* '''Redirect Target IP''': Your mail server IP (here in our example it’s <code>192.168.5.3</code>)
#* '''Redirect Target Port''': 25
#* '''Description''': “Receive Emails”
# '''Important''': Check “Add associated filter rule”
# Click '''Save'''
# Click '''Apply Changes'''

'''Critical Note''': Port 25 MUST be open or you’ll never receive email. This is non-negotiable for a mail server.

<blockquote>'''NOTE:''' When setting up port forwarding for a mail server, make sure that your ISP isn’t blocking it to stop spam. Yours might. It’s not unheard of with residential internet providers. You are paying for a residential connection, not a business one, and they’ll [https://www.youtube.com/watch?v=izXnCkrfjO0 remind you of it way they can](actually, they’ll do that even when you pay $409.99/mo for the business one).
</blockquote>
<span id="step-6-add-pfsense-firewall-rules-for-real"></span>
== Step 6: Add pfSense Firewall Rules (for real) ==

You don’t need to add ALL these rules below. If you are okay with being connected to your VPN, or on your local network, to receive &amp; send email, the only rule you need to add is rule #1 so you can receive mail which you just did.

If you want to allow IP addresses that are NOT connecting to your server via VPN into your mail server, you would create an alias with those IPs using the steps in Lesson 1 above, and then use that alias (called <code>mailserver_trusted_clients</code> in this case) for everything.

One instance would be if you use a service like '''Freshdesk''' for customer service &amp; opt to use your own mail server. In this case, you would have to [https://support.freshdesk.com/support/solutions/articles/50000005619-allowlist-nat-ips '''allow their IP addresses to access your server'''] so that Freshdesk can read your customer service inbox, and send emails as your customer service email.

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:lu55028jxdmy_tmp_1f1b2c6a.png
File:lu55028jxdmy_tmp_31037c49.png
File:lu55028jxdmy_tmp_8d77cc05.png
File:lu55028jxdmy_tmp_54c8a9f1.png
File:lu55028jxdmy_tmp_17e7dded.png
File:lu55028jxdmy_tmp_cc025f84.png
</gallery>

<span id="rule-1-forwarding-smtp-port-25-the-only-rule-you-need-if-you-are-using-openvpn-to-connect-to-your-mailserver"></span>
=== Rule 1: Forwarding SMTP (Port 25) – the ONLY rule you need if you are using OpenVPN to connect to your mailserver! ===

* '''Protocol''': IPv4 TCP
* '''Source''': Any
* '''Destination''': 192.168.5.3
* '''Port''': 25 (SMTP)
* '''Description''': NAT Forward Postfix SMTP to Mailcow

'''What this rule does:'''

* This rule forwards unsecured SMTP traffic on port 25 to the Mailcow server at 192.168.5.3.
* SMTP on port 25 is traditionally used for sending emails between email servers. However, it’s not encrypted by default, meaning the data can be sent in plain text.
* '''Why this is ALWAYS needed''': Although not as secure as SMTPS, port 25 is required for email delivery between servers on the internet. When your Mailcow server sends or receives emails from other email servers, it typically uses SMTP on port 25. This rule makes sure that your Mailcow server can communicate with other email servers to handle incoming and outgoing email traffic. Keeping port 25 closed means saying goodbye to receiving email. If you’re like me, this might be step 1 to solving a lot of life’s problems…

<span id="rule-2-forwarding-smtps-port-465"></span>
=== Rule 2: Forwarding SMTPS (Port 465) ===

* '''Protocol''': IPv4 TCP
* '''Source''': <code>mailserver_trusted_clients</code>
* '''Destination''': 192.168.5.3
* '''Port''': 465 (SMTP/S)
* '''Description''': NAT Forward Postfix SMTPS to Mailcow

<blockquote>'''What this rule does''': - This rule allows secure SMTP (SMTPS) traffic on port 465 from the clients defined in the <code>mailserver_trusted_clients</code> alias to be forwarded to the Mailcow server running on 192.168.5.3. For instance, if you are integrating self-hosted-email with a service like [https://support.freshdesk.com/support/solutions/articles/195170-using-custom-email-servers-to-set-up-support-emails freshdesk], you would want to open this port so their app can send emails using your server. However, you would not want to open it to the entire world, just for the clients you want. In the case of freshdesk, you might make a <code>mailserver_trusted_clients</code> alias with all of [https://support.freshdesk.com/support/solutions/articles/50000005619-allowlist-nat-ips freshdesk’s IP addresses] so they make it through on port 465, but nobody else does. - SMTP (Simple Mail Transfer Protocol) is the protocol used for sending emails. The S at the end of SMTPS indicates that this is a secure version of SMTP, meaning the communication is encrypted using SSL/TLS. - '''When this is needed''': This rule allows email clients that are NOT connected to your server via VPN to send emails using encryption. If this port is closed, they will not be able to connect to your mail server to send mail. - '''When this NOT needed''': This rule is unnecessary if you are sending mail by connecting to your mailserver via VPN, or locally on your home network. It is unnecessary if you do not have external services such as freshdesk that you integrate with your mailserver.
</blockquote>
<span id="rule-3-forwarding-submission-port-587"></span>
=== Rule 3: Forwarding Submission (Port 587) ===

* '''Protocol''': IPv4 TCP
* '''Source''': <code>mailserver_trusted_clients</code>
* '''Destination''': 192.168.5.3
* '''Port''': 587 (SUBMISSION)
* '''Description''': NAT Forward Postfix Submission to Mailcow

<blockquote>'''What this rule does''': - This rule forwards traffic on port 587 to your Mailcow server at 192.168.5.3.
</blockquote>
* Port 587 is used for email submission by clients (i.e., when you’re sending an email through an email client like Outlook or Thunderbird). This port requires authentication and typically uses STARTTLS to secure the connection.
* '''Why this is needed''': Unlike port 25 (which is often used for server-to-server email transmission), port 587 is specifically used for sending emails from a client to the server. When you configure an email client to send messages, you often use port 587 with authentication. This rule makes sure that clients (in this case, the trusted clients defined in <code>mailserver_trusted_clients</code>) can securely submit their emails for sending through Mailcow.
* '''When this NOT needed''': This rule is unnecessary if you are sending mail by connecting to your mailserver via VPN, or locally on your home network. It is unnecessary if you do not have external services such as freshdesk that you integrate with your mailserver

<span id="rule-4-forwarding-imap-port-143"></span>
=== Rule 4: Forwarding IMAP (Port 143) ===

* '''Protocol''': IPv4 TCP
* '''Source''': <code>mailserver_trusted_clients</code>
* '''Destination''': 192.168.5.3
* '''Port''': 143 (IMAP)
* '''Description''': NAT Forward Dovecot IMAP to Mailcow

<span id="what-this-rule-does"></span>
==== What this rule does: ====

* This rule forwards IMAP traffic on port 143 to the Mailcow server at 192.168.5.3.
* IMAP (Internet Message Access Protocol) is used by email clients to retrieve emails from the mail server. IMAP allows users to keep their emails on the server and access them from multiple devices.
* '''Why this is needed''': This rule allows clients to access their emails using the non-encrypted version of IMAP on port 143. It allows clients to view and manage their emails stored on the server without downloading them to their devices.
* '''When this NOT needed''': This rule is unnecessary if you are receiving mail by connecting to your mailserver via VPN, or locally on your home network. It is unnecessary if you do not have external services such as freshdesk that you integrate with your mailserver

<span id="rule-5-forwarding-imaps-port-993"></span>
=== Rule 5: Forwarding IMAPS (Port 993) ===

* '''Protocol''': IPv4 TCP
* '''Source''': <code>mailserver_trusted_clients</code>
* '''Destination''': 192.168.5.3
* '''Port''': 993 (IMAP/S)
* '''Description''': NAT Forward Dovecot IMAPS to Mailcow

<span id="what-this-rule-does-1"></span>
==== What this rule does: ====

* This rule forwards secure IMAP traffic (IMAPS) on port 993 to the Mailcow server.
* IMAPS is the encrypted version of IMAP. It uses SSL/TLS to secure communication between the email client and the server.
* '''Why this is needed''': This rule allows users to securely access their emails stored on the server using IMAP. This is the preferred method for most modern email clients, as it encrypts the communication, making sure that sensitive information like email contents and credentials are protected while being retrieved by the client.
* '''When this NOT needed''': This rule is unnecessary if you are receiving mail by connecting to your mailserver via VPN, or locally on your home network. It is unnecessary if you do not have external services such as freshdesk that you integrate with your mailserver

<span id="rule-6-forwarding-pop3-port-110"></span>
=== Rule 6: Forwarding POP3 (Port 110) ===

* '''Protocol''': IPv4 TCP
* '''Source''': <code>mailserver_trusted_clients</code>
* '''Destination''': 192.168.5.3
* '''Port''': 110 (POP3)
* '''Description''': NAT Forward Dovecot POP3 to Mailcow

<span id="what-this-rule-does-2"></span>
==== What this rule does: ====

* This rule forwards POP3 traffic on port 110 to the Mailcow server.
* POP3 (Post Office Protocol version 3) is another protocol used to retrieve emails from the server. Unlike IMAP, POP3 typically downloads emails to the local device and removes them from the server.
* '''Why this is needed''': This rule allows clients to retrieve emails using POP3. Some users or legacy email clients may prefer to use POP3 if they want to download and store emails locally rather than keeping them on the server.
* '''When this NOT needed''': This rule is unnecessary if you are receiving mail by connecting to your mailserver via VPN, or locally on your home network. Also, why are you even thinking of using POP3? Don’t do this.

<span id="rule-7-forwarding-pop3s-port-995"></span>
=== Rule 7: Forwarding POP3S (Port 995) ===

* '''Protocol''': IPv4 TCP
* '''Source''': <code>mailserver_trusted_clients</code>
* '''Destination''': 192.168.5.3
* '''Port''': 995 (POP3/S)
* '''Description''': NAT Forward Dovecot POP3S to Mailcow

<span id="what-this-rule-does-3"></span>
==== What this rule does: ====

* This rule forwards secure POP3 (POP3S) traffic on port 995 to the Mailcow server.
* POP3S is the encrypted version of POP3, using SSL/TLS for secure communication.
* '''Why this is needed''': This rule enables users to securely retrieve their emails using POP3S. This is preferred over regular POP3 because it makes sure that the email contents and credentials are transmitted securely.
* '''When this NOT needed''': This rule is unnecessary if you are receiving mail by connecting to your mailserver via VPN, or locally on your home network. Also why are you even thinking of using POP3? Don’t do this. Use IMAP, POP3 in 2024 is pure insanity.

<span id="rule-8-forwarding-managesieve-port-4190"></span>
=== Rule 8: Forwarding ManageSieve (Port 4190) ===

* '''Protocol''': IPv4 TCP
* '''Source''': <code>mailserver_trusted_clients</code>
* '''Destination''': 192.168.5.3
* '''Port''': 4190
* '''Description''': NAT Forward Dovecot ManageSieve to Mailcow

<span id="what-this-rule-does-4"></span>
==== What this rule does: ====

* This rule forwards ManageSieve traffic on port 4190 to the Mailcow server.
* ManageSieve is a protocol used to manage server-side email filtering rules (such as automated sorting of emails into folders, marking emails as spam, etc.). This is done on the server side rather than through a client-side rule.
* '''Why this is needed''': This rule allows trusted clients to create and manage email filtering rules on the server. For example, users can create rules to automatically move incoming emails from a certain sender into a specific folder. It’s useful for managing email organization and automating tasks at the server level. I don’t bother with this, but you can if you want to.

<span id="tldr-of-self-hosted-email-firewall-rules"></span>
== TL;DR of self-hosted email firewall rules: ==

<span id="using-openvpn-to-connect-to-your-mailserver"></span>
=== Using OpenVPN to connect to your mailserver? ===

Port 25 is all you have to open to the public so you receive mail from other servers.

<span id="need-clients-outside-lan-that-dont-have-vpn-access-to-connect-to-your-mailserver"></span>
=== Need clients outside LAN that don’t have VPN access to connect to your mailserver? ===

Then you gotta make an alias with their IPs &amp; make all of the rules I provided above.

Let’s say you want ANY IP from ANYWHERE IN THE WORLD to connect to your mailserver; which is a horrible idea; instead of an alias, you’d specify “any” in the “source” section.

This is a bad idea, IMO, on par with the bad idea of being a newbie &amp; doing self-hosted mail.

<blockquote>'''What you should do:''' Just stick to using a VPN to access your inbox, install OpenVPN &amp; K9 Mail on your Android phone and be done with it. Connecting to your VPN on a laptop as well is very easy, it’s one click or one command in the terminal &amp; you should be doing that so you can access all of your other services anyway.**
</blockquote>
<span id="port-25-smtp"></span>
=== Port 25 (SMTP) ===

* '''Why it is open to everyone''': Port 25 is used for server-to-server email transmission, which means email servers from around the world need to be able to reach your Mailcow server to deliver incoming mail. Since this is a very important function for your mail server, it makes sense to allow traffic on port 25 from any source.
* '''Security concerns''': Since port 25 is open to the world, it can be targeted by spammers or malicious actors trying to exploit the service. However, this is mitigated by using tools such as <code>fail2ban</code>, <code>rspamd</code>, and strong SMTP authentication policies to detect and block abuse.

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:lu55028jxdmy_tmp_d4378b80.png
File:lu55028jxdmy_tmp_7ea57844.png
File:lu55028jxdmy_tmp_9d6a661d.png
File:lu55028jxdmy_tmp_279c986f.png
File:lu55028jxdmy_tmp_84524b73.png
File:lu55028jxdmy_tmp_6660b4ba.png
File:lu55028jxdmy_tmp_9116781d.png
File:lu55028jxdmy_tmp_a783a2bb.png
File:lu55028jxdmy_tmp_e9001ea9.png
File:lu55028jxdmy_tmp_690c5265.png
File:lu55028jxdmy_tmp_a8761f8d.png
</gallery>

<span id="step-7-verify-smtp-relay-setup"></span>
== Step 7: Verify SMTP Relay Setup ==

# '''Test Email Delivery''':
# Once the configuration is saved, send a test email to ensure Mailcow is using Postmark to relay emails successfully. I would suggest sending your test email to four addresses:
#* Email to yourself (same email in Mailcow you are sending from).
#* Email to another mailbox on Mailcow.
#* Email to a “friendly” server, i.e., something not hosted by the main mega providers (another person who hosts their own email).
#* A Gmail/iCloud/Microsoft email address.

Each one tests a portion of the chain.

* If 1 doesn’t work, you’re hopelessly screwed.
* If 2 works but not 3, perhaps a network problem.
* If 1, 2, &amp; 3 work but not 4, you’ve likely screwed up something in the SMTP relay or DNS records process, but the networking configuration and Mailcow setup in general is mostly working. It’s also possible that you did everything right, but Google/Apple/Microsoft still hate you. It’s ok. You can’t hate them back though. As my first studio employer told me, ''“Louis, you hate nothing, you intensely dislike it!”''

If all 4 work, great! If you get something like this in your email when sending, you made a stupid typo when setting up SMTP relay. Can you find mine?

<pre> &gt; This is the mail system at host mail.louishomeserver.chickenkiller.com.
  &gt; I'm sorry to have to inform you that your message could not be delivered to one or more recipients. It's attached below. For further assistance, please send mail to postmaster. If you do so, please include this problem report. You can delete your own text from the attached returned message.
  &gt; The mail system
  &gt; &lt;rossmanngroup@gmail.com&gt;: Host or domain name not found. Name service error for name=smtp.postmark.com type=A: Host not found
  &gt; &lt;louis@rossmanngroup.com&gt;: Host or domain name not found. Name service error for name=smtp.postmark.com type=A: Host not found</pre>
This concludes the guide on setting up Postmark as an SMTP relay for your Mailcow server, configuring DNS records, and setting up firewall rules. Remember to double-check all your configurations and test thoroughly to provide everything is working as expected. Or, don’t &amp; give up. The latter is recommended.

<span id="step-8-spam-controls"></span>
== Step 8 – Spam controls ==

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:lu55028jxdmy_tmp_e4a0e1aa.png
File:lu55028jxdmy_tmp_4909c297.png
File:lu55028jxdmy_tmp_83cfe68f.png
File:lu55028jxdmy_tmp_a210ae1a.png
</gallery>

<span id="accessing-the-rspamd-interface"></span>
==== Accessing the Rspamd Interface ====

To access the Rspamd web interface, you need to be logged in as an administrator on Mailcow. Here’s how you do it:

# Go to <code>http://your-mailcow-address/admin</code>
# Enter your admin password
# Navigate to '''System &gt; Configuration &gt; Actions &gt; Rspamd'''
# Set your password for Rspamd

Once you’re in, you can train the system manually and upload things for it to learn from.

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:lu55028jxdmy_tmp_3a6b6ed2.png
File:lu55028jxdmy_tmp_bd3e5f50.png
File:lu55028jxdmy_tmp_5792c482.png
</gallery>

<span id="accessing-your-inboxs-spam-controls"></span>
==== Accessing YOUR inbox’s spam controls ====

# Log into the Mailcow interface with your EMAIL USERNAME &amp; PASSWORD, NOT AS ADMIN
# Go to '''Email → Spam Filter'''
# Slide the slidy thingy &amp; have fun :)

To set the spam controls for your specific account, log in as your USER to the web interface, not an admin.

<span id="pfblockerng-for-spam-prevention"></span>
==== pfBlockerNG for spam prevention ====

Remember when we set up '''pfBlockerNG''' in our '''pfSense''' router?

'''pfBlockerNG''' has IPv4 blocklists like Lashback that are great for reducing spam from known bad actors, such as people who explicitly send email to addresses that they know are on ''“unsubscribe”'' lists. If you use '''pfBlockerNG''' with these lists, when servers with IPs on these blocklists try to send you mail on port 25, they will be blocked ''at the router level'' before these known bad actors even make their way to your <code>mailcow</code> server or spam filter.

Take a look at these lists. They are incredibly useful!

<span id="dont-do-this"></span>
== Don’t do this ==

'''Warning:''' Self-hosting email is a high-maintenance, complicated task. Just because you can do it doesn’t mean you should. It’s a decision you might regret later.

<span id="home-assistant-to-control-your-air-conditioners-full-smarthome-control"></span>
= Home Assistant to control your air conditioners &amp; full smarthome control =

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:lu55028jxdmy_tmp_5994d652.png
File:lu55028jxdmy_tmp_fc2a8f08.png
File:lu55028jxdmy_tmp_d9a40228.png
File:lu55028jxdmy_tmp_de21b07.png
</gallery>

<span id="what-is-home-assistant"></span>
== What is Home Assistant? ==

Home Assistant allows you to control everything from your lights to your air conditioner to your car’s remote start, all within an open-source system that YOU control! It is a system that works with plugins developed by open-source devs around the world who are just as frustrated as you are that the smart home future we were promised is chock full of spyware, subscriptions, and enshittification. We’re going to be using this to adjust an air conditioner’s temperature, so if we’re going to be home early, we can tell it to turn on remotely a little earlier without allowing the A/C to connect to the internet, and also for getting alerts when someone walks by one of our security cameras.

<span id="step-1-installing-home-assistant"></span>
== Step 1: Installing Home Assistant ==

<span id="download-the-home-assistant-kvm-image-and-prepare-it-for-use"></span>
==== 1.1 Download the Home Assistant KVM Image and Prepare it for Use ====

<ol style="list-style-type: decimal;">
<li><p>Go to the ''[https://www.home-assistant.io/installation/linux official Home Assistant website]''.</p></li>
<li><p>'''Find the KVM Image''':</p>
<ul>
<li><p>Scroll down to the section titled “KVM(virt-manager)”.</p></li>
<li><p>Click the link to download the latest KVM <code>.qcow2.xz</code> image from the official Home Assistant GitHub releases. Alternatively, you can download directly from the GitHub link provided ''[https://github.com/home-assistant/operating-system/releases/download/13.1/haos_ova-13.1.qcow2.xz here]''.</p></li>
<li><p>''(Note: This file version will change over time, so make sure you are downloading the latest release.)''</p></li>
<li><p>''MAKE SURE YOU GRAB THE ONE FOR KVM VIRSH VIRTUAL MACHINE MANAGER IN LINUX, NOT THE VIRTUALBOX ONE!''</p></li></ul>
</li>
<li><p>'''Download and Unzip the Image''':</p>
<ul>
<li><p>Once the download is complete, you’ll need to unzip the <code>.qcow2.xz</code> file. Run the following command to decompress the file:</p>
<pre>xz -d haos_ova-13.1.qcow2.xz</pre></li>
<li><p>''(Make sure the filename reflects the version you downloaded, as it may vary.)''</p></li></ul>
</li>
<li><p>'''Move the Unzipped Image to the Correct Directory''':</p>
<ul>
<li><p>Move the decompressed <code>.qcow2</code> file to your virtual machine images directory, typically <code>/var/lib/libvirt/images/</code>. Use the following command to move it:</p>
<pre>sudo mv ~/Downloads/haos_ova-13.1.qcow2 /var/lib/libvirt/images/</pre></li></ul>
</li>
<li><p>'''Set the Correct Ownership and Permissions''':</p>
<ul>
<li><p>Change the ownership of the image file so that it is owned by the <code>libvirt-qemu</code> user and group:</p>
<pre>sudo chown libvirt-qemu:libvirt /var/lib/libvirt/images/haos_ova-13.1.qcow2</pre></li></ul>
</li>
<li><p>'''Set the right permissions to make sure it is readable and writable by the owner, but not everyone else:'''</p></li></ol>

<pre>sudo chmod 0640 /var/lib/libvirt/images/haos_ova-13.1.qcow2</pre>
<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:lu55028jxdmy_tmp_6a102293.png
File:lu55028jxdmy_tmp_fc378926.png
File:lu55028jxdmy_tmp_7540e9f2.png
File:lu55028jxdmy_tmp_c44a31fb.png
File:lu55028jxdmy_tmp_83761502.png
File:lu55028jxdmy_tmp_5af2cd09.png
File:lu55028jxdmy_tmp_696b6c09.png
File:lu55028jxdmy_tmp_48c367ce.png
</gallery>

<span id="install-the-home-assistant-virtual-machine-on-ubuntu-server-linux"></span>
==== 1.2 Install the Home Assistant Virtual Machine on Ubuntu Server Linux ====

Before, we chose “local install media” when installing Ubuntu Server to our virtual machine for mailcow, but Home Assistant is a little different. It’s an operating system that is all ready to go – it’s installed, configured, etc. We are going to be choosing the “import existing disk image” option to boot it up.

* '''Open Virtual Machine Manager''':
** Right-click on the desktop of your Ubuntu Server.
** Navigate to '''Applications &gt; System &gt; Virtual Machine Manager'''.
* '''Create a New Virtual Machine''':
** Once Virtual Machine Manager is open, click on '''Create a new virtual machine'''.
** In the wizard that appears, choose the option '''Import existing disk image'''.
** Unlike the past virtual machine where we were installing an operating system from scratch, this is an image of an operating system that has already been “installed” and configured elsewhere; therefore, all we need to do is import it.
* '''Select the Home Assistant Image''':
** When prompted to choose an installation source, browse to <code>/var/lib/libvirt/images/</code> and select the Home Assistant <code>.qcow2</code> image you moved in the previous step.
* '''Choose Operating System Type''':
** Select '''Generic Linux 2022''' as the operating system type. The official Home Assistant instructions suggest using a “generic” Linux option.
* '''Set Memory and CPU Allocation''':
** Set the RAM to 2048 MB (2 GB).
** Assign 2 CPUs to the virtual machine.
** It is recommended to use 2 CPUs, even though this might feel like overkill for a thermostat-related function. And it does.
* '''Name the Virtual Machine''':
** In the same setup window, name the virtual machine <code>homeassistant</code>.
* '''Customize Configuration Before Installation''':
** Before clicking '''Finish''', make sure you check the box that says '''Customize configuration before install'''.
* '''Set Firmware to UEFI''': you want UEFI x86-64: <code>/usr/share/OVMF/OVMF_CODE_4M.fd</code> – DO NOT CHOOSE THE ONE THAT SAYS “SECBOOT”

'''Set up this virtual machine to start every time the host computer, happycloud, boots by typing this into a terminal:'''

<pre>virsh autostart homeassistant</pre>
* Check that this is set up properly by typing <code>virsh dominfo homeassistant</code> and seeing if the autostart line is set to enable.
* If you don’t do this, you will realize once it is too late &amp; you’ve left your house after you have rebooted your server (for whatever reason) that none of your services are working. This will suck.

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:lu55028jxdmy_tmp_48c367ce.png
File:lu55028jxdmy_tmp_1f366dc.png
File:lu55028jxdmy_tmp_8ebe041d.png
File:lu55028jxdmy_tmp_d116f2c3.png
File:lu55028jxdmy_tmp_2cfe4d00.png
File:lu55028jxdmy_tmp_9f65df5c.png
File:lu55028jxdmy_tmp_c95bffa.png
File:lu55028jxdmy_tmp_d42cf5c1.png
File:lu55028jxdmy_tmp_f0ebdd74.png
</gallery>

<span id="step-2-start-and-configure-home-assistant"></span>

== Step 2: Start and Configure Home Assistant ==

<span id="start-the-virtual-machine"></span>
==== 2.1 Start the Virtual Machine ====

* In '''Virtual Machine Manager''', locate your Home Assistant virtual machine and start the VM.
* Wait for the machine to boot up fully.

<span id="identify-the-ip-address"></span>
==== 2.2 Identify the IP Address ====

* Once the virtual machine has finished booting, check the console within '''Virtual Machine Manager'''. You will see an IP address displayed (e.g., 192.168.5.16).
* We did not “install” this operating system like with the previous mailcow mailserver installation; we imported someone else’s installation. This means we do not have its IP address, nor were we given an opportunity to set up Home Assistant with a static IP yet. It is good to pay attention here so you see its IP address and know where to find it.
* When it says the URL is ''[http://homeassistant.local/ http://homeassistant.local]'':8123, this is wrong. It is assuming that our “domain” is <code>.local</code>. By default, '''pfSense''' sets this to <code>home.arpa</code>.
* If no IP address is displayed on the console, you can also check your DHCP server or router (like '''pfSense''') to find the IP assigned to the Home Assistant VM.

<span id="access-home-assistant-web-interface"></span>
==== 2.3 Access Home Assistant Web Interface ====

* Open a web browser on your local machine.
* In the address bar, type the following to access the Home Assistant web interface: <code>http://homeassistant.home.arpa:8123</code> (For example: <code>http://192.168.5.16:8123</code>).
* At the time of writing this guide, Home Assistant will only load on <code>http://</code> by default when first started, not <code>https://</code>, if you use their fully-fledged HaOS virtual machine image. Don’t worry, you didn’t break anything.

<span id="follow-on-screen-setup-instructions"></span>
==== 2.4 Follow On-Screen Setup Instructions ====

* It will tell you to wait up to 20 minutes to load.
* You will be greeted by the Home Assistant setup wizard. Follow the on-screen instructions to complete the setup.
* Create a Home Assistant Account: Enter a username, password, and any additional information required.
* Configure Location &amp; Units: Choose your location and preferred units (imperial or metric).
* Add Devices and Services: Home Assistant will begin searching for devices on your network. Depending on your network configuration, devices may automatically be discovered. This is pretty cool. I like this.
* You don’t have to “trust” them, it’s open source so you can see what it is doing while probing. This is not probing to mess with or spy on you, it’s doing this to try to make your life easier… The thing technology was supposed to do for you.

<span id="complete-setup"></span>
==== 2.5 Complete Setup ====

* Once you’ve created your account and finished the basic configuration, Home Assistant will finalize the installation and setup. You are now ready to take back your air conditioner from the proprietary cloud.

<span id="step-3-configure-home-assistant-with-a-static-ip"></span>
== Step 3: Configure Home Assistant with a Static IP ==

<span id="home-assistant-network-configuration"></span>
=== Home Assistant Network Configuration: ===

<div class="figure">

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:lu55028jxdtp_tmp_8dbeaafa.png
</gallery>

</div>
<div class="figure">

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:lu55028jxdtp_tmp_68d29356.png
</gallery>

</div>
<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:lu55028jxdtp_tmp_25b69769.png
</gallery>

<span id="access-home-assistants-network-settings"></span>
==== 3.1 Access Home Assistant’s Network Settings ====

* Open the Home Assistant web interface by navigating to <code>http://[your_homeassistant_ip]:8123</code>.
* Once logged in, go to '''Settings''' (found at the bottom left of the sidebar).
* From the '''Settings''' page, click on '''System''' and then select '''Network'''.

<span id="modify-network-interface"></span>
==== 3.2 Modify Network Interface ====

* In the '''Network''' section, find the network interface (e.g., <code>eth0</code>) that Home Assistant is using.
* Click '''Configure''' next to the interface to edit its settings.

<span id="switch-to-a-static-ip-configuration"></span>
==== 3.3 Switch to a Static IP Configuration ====

* Change the network type from '''DHCP''' to '''Static''' to manually configure the IP address.
* Set the following details:
** '''IP Address:''' Enter the desired static IP address (e.g., <code>192.168.5.4</code>).
** '''Gateway:''' Enter the gateway IP address, the IP of your pfSense router (e.g., <code>192.168.5.1</code>).
** '''DNS Server:''' Enter the IP address of the DNS server (your pfSense router’s IP, e.g., <code>192.168.5.1</code>).

<span id="save-the-configuration"></span>
==== 3.4 Save the Configuration ====

* Once you’ve set the static IP, gateway, and DNS, click '''Save''' to apply the changes.
* Home Assistant will now be reachable at the static IP address you configured.

<span id="save-the-configuration-1"></span>
==== 3.5 Save the Configuration ====

* Once you’ve set the static IP, gateway, and DNS, click '''Save''' to apply the changes.
* Home Assistant will now be reachable at the static IP address you configured.

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:lu55028jxdtp_tmp_fb68028c.png
</gallery>

![(images/lu55028jxdtp_tmp_5504653d.png)

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:lu55028jxdtp_tmp_22a2f148.png
File:Timeline%201_04_41_14_33.png
File:lu55028jxdtp_tmp_b09fba6a.png
</gallery>

<span id="add-a-static-ip-mapping-in-pfsense"></span>
=== Add a Static IP mapping in pfSense ===

<span id="log-in-to-pfsense"></span>
==== 3.6 Log in to pfSense ====

* Open your web browser and navigate to the pfSense web interface (e.g., <code>https://192.168.5.1</code> or <code>https://pfSense.home.arpa</code>).
* Log in using your admin credentials.

<span id="navigate-to-dhcp-server-settings"></span>
==== 3.7 Navigate to DHCP Server Settings ====

* Once inside pfSense, go to '''Services &gt; DHCP Server'''.
* In the DHCP Server settings, go to the '''LAN''' tab, as this is where you’ll configure the static mapping for devices on your local network.

<span id="add-a-static-ip-mapping"></span>
==== 3.8 Add a Static IP Mapping ====

* Scroll down to the '''DHCP Static Mappings''' section and click on '''Add Static Mapping'''.

<span id="enter-the-information"></span>
==== 3.9 Enter the Information ====

* '''MAC Address:''' Find the MAC address of your Home Assistant virtual machine. To do this:
** In pfSense, navigate to '''Diagnostics &gt; ARP Table'''.
** Look for the MAC address associated with the Home Assistant VM’s current IP (this can also be found within the Virtual Machine Manager or via the Home Assistant network settings).
* '''IP Address:''' Enter the static IP address you configured earlier in Home Assistant (e.g., <code>192.168.5.4</code>).
* '''Description:''' Enter a description for easy identification (e.g., homeassistant).
* '''Hostname:''' Enter <code>homeassistant</code>

<span id="save-and-apply-changes"></span>
==== 3.92 Save and Apply Changes ====

* Click '''Save''' to add the static mapping.
* After saving, click '''Apply Changes''' to make sure the static IP reservation is applied on your network.

<span id="make-sure-this-actually-works"></span>
==== 3.94. Make Sure This Actually Works ====

* After configuring the static IP and DHCP mapping:
** Make sure Home Assistant is reachable at the assigned IP (e.g., <code>http://192.168.5.4:8123</code>).
** In pfSense, you can check the '''Status &gt; DHCP Leases''' section to confirm that Home Assistant is using the correct IP address and that the static mapping is working.

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:lu55028jxdtp_tmp_35f2d05.png
File:lu55028jxdtp_tmp_46f57247.png
File:lu55028jxdtp_tmp_134fbcd6.png
File:lu55028jxdtp_tmp_132130f0.png
File:lu55028jxdtp_tmp_9865e7ff.png
File:lu55028jxdtp_tmp_20521c00.png
File:lu55028jxdtp_tmp_dcbab263.png
File:lu55028jxdtp_tmp_8e2628b5.png
File:lu55028jxdtp_tmp_78c8cb71.png
</gallery>

<span id="step-4-set-up-the-venstar-thermostat-so-home-assistant-can-see-it"></span>
== Step 4: Set Up the Venstar Thermostat so Home Assistant can see it ==

<span id="connect-the-venstar-thermostat-to-wi-fi"></span>
==== 4.1. Connect the Venstar Thermostat to Wi-Fi ====

* On the thermostat, go to '''Wi-Fi Setup'''. The thermostat will display a list of available networks.
* Select your desired Wi-Fi network and enter the password if necessary.
* Once connected, make sure that the thermostat remains on the same network that your Home Assistant instance is on, or another network that can communicate with Home Assistant.

<blockquote>'''NOTE:''' Make sure you tap the right network; this garbage touchscreen makes it very easy to tap the wrong network &amp; not notice it. Whoever chose this touchscreen should be in the same prison with the engineers of the A1237/A1304 model MacBook Air from 2008.
</blockquote>
<span id="configure-the-local-api-on-the-thermostat"></span>
==== 4.2 Configure the Local API on the Thermostat ====

* On the thermostat, navigate to the '''Local API Options'''.
* Turn on '''Local API access''', which is necessary for Home Assistant to communicate with the thermostat.
* Set a username (e.g., second floor), and configure a Basic Auth password. You’ll need this information when adding the thermostat in Home Assistant.

<span id="assign-a-static-ip-to-the-thermostat"></span>
==== 4.3 Assign a Static IP to the Thermostat ====

* On the thermostat, navigate to '''Manual Setup &gt; Network Settings'''.
* Assign a static IP to the thermostat. This ensures that the IP address does not change, which is very important or you will find yourself freezing to death when you can’t turn off the A/C.

<blockquote>'''NOTE:''' Home assistant needs to know where to find the thermostat; at the same place, every single time. Anytime you attach an IoT device to your network, it is a good practice to give it a static IP. You will find out later in the “syncthing” section why expecting “auto locate” features to work proprerly is a bad idea. Summers are 117 degrees fahrenheit in Texas; I’m not trusting that to DHCP.
</blockquote>
* '''IP Address:''' 192.168.5.18 (or another appropriate IP in your network range)
* '''Gateway:''' 192.168.5.1 (typically your '''pfSense''' router’s IP)
* '''DNS Server:''' 192.168.5.1
* '''Subnet Mask:''' 255.255.255.0

<span id="confirm-the-settings"></span>
==== 4.4 Confirm the Settings ====

* After entering the network configuration, make sure that the thermostat is connected and reachable on your network.

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:lu55028jxdtp_tmp_3756eeae.png
File:lu55028jxdtp_tmp_25adf640.png
File:lu55028jxdtp_tmp_a95a89f4.png
File:lu55028jxdtp_tmp_28b42c93.png
File:lu55028jxdtp_tmp_ff730254.png
File:lu55028jxdtp_tmp_83532402.png
File:lu55028jxdtp_tmp_8b9bc651.png
File:lu55028jxdtp_tmp_5e9c6f46.png
File:lu55028jxdtp_tmp_437e4496.png
File:lu55028jxdtp_tmp_f1dcce3c.png
File:lu55028jxdtp_tmp_195f5ef1.png
File:lu55028jxdtp_tmp_9badefce.png
File:lu55028jxdtp_tmp_63c458ef.png
File:lu55028jxdtp_tmp_69ea9520.png
</gallery>

<span id="step-5-add-the-venstar-integration-in-home-assistant"></span>
== Step 5: Add the Venstar Integration in Home Assistant ==

<span id="access-home-assistant"></span>
==== 5.1 Access Home Assistant ====

* Open the Home Assistant web interface by navigating to <code>http://[your_homeassistant_ip]:8123</code>.
* Log in with your Home Assistant credentials.

<span id="navigate-to-the-integrations-section"></span>
==== 5.2 Navigate to the Integrations Section ====

* In Home Assistant, click on '''Settings''' from the sidebar.
* Under '''Settings''', go to '''Devices &amp; Services'''.
* Click on '''Add Integration'''.

<span id="search-for-the-venstar-integration"></span>
==== 5.3 Search for the Venstar Integration ====

* In the search bar, type '''Venstar''' to find the Venstar integration.

<span id="enter-thermostat-details"></span>
==== 5.4 Enter Thermostat Details ====

* When prompted, enter the following information:
** '''Host:''' Enter the static IP address you assigned to the thermostat (e.g., 192.168.5.18).
** '''Username:''' Enter the username you set up on the thermostat (e.g., second floor).
** '''PIN Code:''' If required by your thermostat model, enter the PIN code (optional). THIS THERMOSTAT DOES NOT REQUIRE PIN
** '''SSL Certificate:''' yes by default for my thermostat, may be different for yours. For mine, it is yes.

<span id="submit-the-integration"></span>
==== 5.5 Submit the Integration ====

* Click '''Submit'''. Home Assistant will now attempt to connect to your Venstar thermostat using the provided details.
* If successful, the Venstar thermostat will be added as a device in Home Assistant.

<span id="step-6-configure-the-thermostat-in-home-assistant"></span>
== Step 6: Configure the Thermostat in Home Assistant ==

<span id="assign-the-thermostat-to-an-area"></span>
==== 6.1 Assign the Thermostat to an Area ====

* After adding the integration, you can assign the thermostat to an area, such as Living Room. This part confuses me, there are so many labels &amp; subcategories. It is easy to get lost in them all.

<span id="add-thermostat-controls-to-your-dashboard"></span>
==== 6.2 Add Thermostat Controls to Your Dashboard ====

* Go to '''Overview''' in Home Assistant.
* Click the diagonal line that is supposed to look like a pencil in the upper right hand corner.
* Now you are in the edit dashboard menu, that does absolutely nothing.
* Click the three dots in the upper right corner, then click '''take control''' so you can actually edit your dashboard.
* Click '''start with an empty dashboard'''
* Click on '''Edit Dashboard''', then click '''Add Card'''.
* Select '''Thermostat''' as the card type, and choose your Venstar thermostat from the list.
* Give the thermostat a cool name, like Second Floor Thermostat, and click '''Done'''.

<span id="customize-the-dashboard"></span>
==== 6.3 Customize the Dashboard ====

* If you want to adjust or hide certain things, you need to click '''“Take Control”''' in what is some of the most confusing UI of all time.

<blockquote>'''NOTE:''' You have to hit '''Take Control''' in order to do anything with the interface. This is not obvious or intuitive. I set up my dashboard on android when I set up my own system, so I never saw the dashboard in the web interface. I tried the web interface dashboard for the first time when I did this guide. It owned me good.
</blockquote>
<span id="use-the-venstar-thermostat-in-home-assistant"></span>
==== 6.4. Use the Venstar Thermostat in Home Assistant ====

# Control the Thermostat
#* From the dashboard, you can now adjust the temperature, set heating or cooling modes, and control the fan (e.g., always on or only when the compressor is active).
# View Historical Data
#* Home Assistant provides historical graphs showing temperature changes and thermostat actions (e.g., target temperature vs. actual temperature) over time, which you can view directly in the thermostat card on your dashboard.

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:lu55028jxdtp_tmp_42249f5f.png
File:lu55028jxdtp_tmp_d9a6710f.png
File:lu55028jxdtp_tmp_1308f505.png
File:lu55028jxdtp_tmp_d1f64c00.png
File:lu55028jxdtp_tmp_c83f5b97.png
File:lu55028jxdtp_tmp_f7a08a29.png
File:lu55028jxdtp_tmp_2cf2e4b1.png
</gallery>

<span id="step-7-install-home-assistant-application-on-your-phone-to-adjust-pos-thermostat-so-you-never-have-to-touch-its-touchscreen-again"></span>
== Step 7: Install Home Assistant Application on Your Phone to Adjust POS Thermostat So You Never Have to Touch Its Touchscreen Again ==

<span id="install-the-home-assistant-app-on-android"></span>
==== 7.1 Install the Home Assistant App on Android ====

# Open the Google Play Store
#* On your Android device, open the '''Google Play Store''' app.
# Search for Home Assistant
#* In the search bar, type '''Home Assistant'''.
# Install the App
#* Once you find the Home Assistant app (from Nabu Casa), tap '''Install''' to download and install it on your phone.
# Open the App
#* After installation is complete, tap '''Open''' to start the Home Assistant app.

<span id="make-sure-openvpn-connect-is-connected"></span>
==== 7.2 Make Sure OpenVPN Connect is Connected ====

* Open the '''OpenVPN Connect''' app and connect to the VPN profile you set up for accessing your home network.

It’s important that you are connected to your VPN when accessing Home Assistant from outside your local network! None of this is set up with open ports to the outside world. Without VPN, no air conditioning for you.

<span id="log-in-to-home-assistant-on-android"></span>
==== 7.3. Log In to Home Assistant on Android ====

# '''Launch the Home Assistant App'''
#* Open the Home Assistant app you installed earlier.
# '''Connect to Home Assistant'''
#* The app may automatically search for your Home Assistant instance. If it doesn’t find it, you can manually enter the IP address. Since you are connected via VPN, you’ll enter your Home Assistant server’s local IP, aka [http://192.168.5.4/ http://192.168.5.4]:8123.
#* You can’t add <code>192.168.5.4</code>.
#* You can’t add <code>192.168.5.4:8123</code>.
#* IT MUST BE [http://192.168.5.4/ http://192.168.5.4]:8123.
#* You have to have the <code>http://</code> and the port.
# '''Log In'''
# '''Enable Location Tracking (Optional)'''
#* You’ll be prompted to enable location tracking. You can choose to allow or deny this depending on your preferences. They’re not spying on you though; they’re nice people, not like the [https://www.texasattorneygeneral.gov/news/releases/attorney-general-ken-paxton-sues-general-motors-unlawfully-collecting-drivers-private-data-and evil bastards] that sold you your car.

<span id="adjust-the-thermostat-using-the-home-assistant-app"></span>
==== 7.4. Adjust the Thermostat Using the Home Assistant App ====

# '''Access the Thermostat in the App'''
#* After logging in, you’ll see the Home Assistant dashboard.
#* FInd your Venstar Thermostat (e.g., Second Floor Thermostat) on the dashboard.
# '''Control the Thermostat'''
#* Tap on the thermostat card to open the controls.
#* From here, you can:
#** '''Adjust the Temperature:''' Use the sliders or buttons to set the temperature.
#** '''Set Mode:''' Change the thermostat to Heat, Cool, or Auto.
#** '''Fan Control:''' Choose whether the fan should run Continuously or only when the heat/AC is on.
# '''Monitor Historical Data on when you Had it on'''
#* The app will display historical data showing the target temperature and current room temperature over time, so you can see when it was on, etc.

<blockquote>Historical data will not show how many times you have punched the thermostat’s touchscreen, cursed at Venstar, or threatened the lives of the people who engineered it. But it should.
</blockquote>
<span id="home-surveillance-camera-system-with-alerts"></span>
= Home surveillance camera system with alerts: =

Next up, I’m going to show you how to set up a home surveillance system. This system will send alerts to your phone whenever someone passes by the cameras around your house. These security cameras use standard protocols like <code>RTSP</code> and <code>ONVIF</code> – they are STANDARDS, and as a result, they cannot be taken away from you later. When you buy these cameras, YOU own the cameras, YOU own the video, and YOU own the alerts system. No cloud subscriptions, nobody having the ability to change the terms of the sale. No bullshit. :)

<span id="step-1-choosing-cameras"></span>
== Step 1: Choosing cameras ==

For this tutorial, I am using a Hikvision camera as an example.

<span id="why-choose-hikvision-cameras"></span>
=== Why Choose Hikvision Cameras? ===

I’m settling with Hikvision for the same reason your parents settled on each other; not because they’re the best, but because they’re good enough &amp; available. These cameras are everywhere, especially in small businesses in New York City. When businesses close and liquidate, you can find these cameras as cheap as $150 for a lot of eight, that do 2 megapixel video in good enough quality to see license plates and make out fine facial features.

You can find these cameras on eBay for as low as $30 or $40 each, and sometimes even cheaper in bulk at liquidation sales. Because they’re so popular, &amp; cheap for the quality you can get, I’m using them as an example.

<span id="alternatives-for-the-best-quality"></span>
=== Alternatives for the Best Quality ===

If you’re looking for the best of the best, I suggest cameras from a company called Axis. They make really high-quality stuff, but you’re not finding a lot of 8 for $150 in a liquidation sale.

If you want the best, there’s nothing like '''AXIS'''.

If you are concerned about Chinese equipment phoning home &amp; sending Xi Jinping photos of you pissing in your backyard at 1 AM, I’ll show you how to create a second network in '''pfSense''' at the end of this guide. Once that’s done, you can make it way more difficult for Xi to get a good view.

<span id="step-2-setting-up-the-hikvision-camera-from-scratch"></span>
== Step 2: Setting up the Hikvision Camera from Scratch ==

<span id="introduction-to-hikvision-ip-issues"></span>
==== 2.1 Introduction to Hikvision IP issues ====

When you get a good camera, it usually uses DHCP to connect to your network. This means when you hook it up, you’ll be able to see it in the ARP table on your '''pfSense''' router. It’ll grab an IP address that your router provides, and boom, it’s on the network.

…I said a GOOD camera. These are (likely grey market) Hikvisions set up into god knows what configuration being sold by a business liquidator.

Cheaper cameras might not do this. They often come with some weird static IP like <code>192.0.0.64</code>, and you have no idea what it’s trying to connect to. Hikvision cameras can be like this sometimes.

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:lu55028jxdtp_tmp_99f3299b.png
File:lu55028jxdtp_tmp_3bd9e222.png
File:lu55028jxdtp_tmp_65220560.png
</gallery>

<span id="download-the-sadp-tool"></span>
==== 2.2 Download the SADP Tool ====

To fix this IP issue, Hikvision offers a tool called SADP. Unfortunately, this tool requires Windows. So, I’m booting up a sandboxed Windows computer here. It’s a burner computer I use for college math classes because, apparently, you can’t learn math on GNU/Linux, so I keep it around for the cancer that is Pearson Vue.

'''Download and Install SADP''': Grab it from [https://www.hikvision.com/us-en/support/tools/hitools/clea8b3e4ea7da90a9/ Hikvision’s website].

Sometimes, these cameras come with passwords that even the seller doesn’t know. You might have to reset it by hitting a button inside the camera to get it back to default settings.

'''Preparing the camera for login'''

Once SADP finds your camera, you can log in and configure it. Often, you’ll need to look up the default password online or in the manual.

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:lu55028jxdtp_tmp_5fb1fcd0.png
File:lu55028jxdtp_tmp_5748d3c3.png
File:lu55028jxdtp_tmp_4f3a2ffe.png
File:lu55028jxdtp_tmp_9ac7122.png
</gallery>

<span id="running-sadp-to-prepare-camera-for-login"></span>
==== 2.3 Running SADP to prepare camera for login ====

Once installed, run SADP and have it find your camera. Once it finds your camera, click on that camera, set it to DHCP, and apply the configuration. You have to enter the password to do this.

The reason we are using DHCP at first rather than static IP is because this is insanely janky &amp; I want to confirm that it even works &amp; lets you log in at all before going further.

If you know the password, you’re done with 99% of the setup. If it doesn’t work, google the default password for that specific model of hikvision camera.

If that doesn’t work, you can either:

* Message the seller and ask them, but 99% of the time they know less than you about whatever they’re liquidating
* Open the camera physically &amp; find a button you can hit to reset it. At that point, the default user/pass you find on google should now work.

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:lu55028jxdtp_tmp_a01aaa48.png
File:lu55028jxdtp_tmp_1803a7a2.png
File:lu55028jxdtp_tmp_56b379c8.png
</gallery>

<span id="logging-into-your-newfound-camera"></span>
==== 2.4 Logging into your newfound camera ====

After this, sign into your '''pfSense''' router and go to '''Status —&gt; DHCP Leases''' to find your camera. I used '''Diagnostics —&gt; ARP Table''' since I’m used to it. Once you know its IP, put it into your web browser and log right in. :)

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:lu55028jxdtp_tmp_8bd543f6.png
File:lu55028jxdtp_tmp_8a8974a8.png
</gallery>

<span id="configuring-a-static-ip"></span>
==== 2.5 Configuring a Static IP ====

First things first, you want to give your camera a static IP address. For instance, if you choose 192.168.5.19, you set it so you always know where to find it. This is necessary; imagine your system goes offline for a few minutes and something steals your camera’s IP address, and now your security camera recorder is trying to get a video feed from your refrigerator? Sadly, by the time this is published, your fridge might actually have a video feed…

* Configure network settings with a static IP:
** Click '''Configuration'''
** Click '''Network''' on the left side
** Uncheck '''DHCP'''
** Set an '''IPv4 Address''' on your subnet, anything from 192.168.5.5-192.168.5.254 will do here.
** Set the '''IPv4 Default Gateway''' to be your '''pfSense''' router.
** Click '''Test''' to make sure you didn’t screw something up before you save this configuration &amp; can no longer log into your camera.
* Set '''Preferred DNS server''' and '''Alternate DNS server''' to the IP address of your '''pfSense''' router, which in our case is 192.168.5.1.
* '''User management''': Set a username and password for security.

<span id="configure-a-static-mapping-in-pfsense"></span>
==== 2.6 Configure a Static Mapping in pfSense ====

Follow the same instructions from our prior static mappings to set up a static mapping for our camera so that other devices do not steal its IP address.

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:lu55028jxdtp_tmp_f9b49c88.png
File:lu55028jxdtp_tmp_94640b52.png
File:lu55028jxdtp_tmp_8384a94b.png
</gallery>

<span id="create-a-real-password-for-the-camera"></span>
==== 2.7 Create a REAL Password for the camera ====

No, we’re not keeping the username and password to “admin/password”

# Once inside the camera’s configuration interface, go to '''Configuration''' at the top.
# Go to '''System''' on the left side.
# Go to '''User Management'''.
# Click '''Modify''' on the admin user.
# Don’t use the word “password” or “12345” as your password.
# Put this in a password manager when you’re done. Not a post-it on your monitor.
# Don’t write the password on the camera. I will come through this screen like Samara from The Ring and drag you so deep down a well you’ll end up on a ''[https://www.youtube.com/@fatal_breakdown cave diving YouTube channel]''.

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:lu55028jxdtp_tmp_a1c6507a.png
</gallery>

<span id="change-video-codec-to-h.264"></span>
==== 2.8 Change Video Codec to H.264 ====

When it comes to video encoding, I’d use H.264 over H.265. '''Frigate''' &amp; web browsers can be fussy playing back H.265, and the quality bump is not something I notice enough to be worth the aggravation. Given this is a beginner’s guide, the safe choice is to use the codec that is less likely to cause aggravation.

'''Frigate''' is going to have two streams – one that detects when something is going on (a dog, a cat, a car, a human, etc.), and another that does the recording. If we have a high-quality stream doing all of the detection work, our system is going to be killing itself all the time unnecessarily. We don’t need 12k Blackmagic Ursa quality video to tell whether we’re looking at a car’s license plate or a plastic bag in the wind. We do need good quality to record, though.

We’re going to set up one high-quality stream for recording, and another lower-quality stream for monitoring what’s going on. This way, we get high-quality video for playback, without unnecessarily blowing up the resource consumption on our computer.

* While logged into the camera interface, click '''Configuration'''.
* Click '''Video/Audio''' on the left side, and select '''Stream Type''' as '''Main Stream (Normal)'''. This is the feed we will be recording.
** For '''Main Stream (Normal)''', set '''Video Encoding''' to '''H.264'''.
** Set '''Video Quality''' to '''Highest'''.
** '''Resolution''' and '''Frame Rate''' are up to you – I like the highest resolution that gets me at least 20 frames per second. Lower than this and it starts to turn into a slideshow.
* Now, select '''Stream Type''' and click onto the 2nd stream listed.
* Set a very low '''Resolution''', something in the 600x300-ish range.
* Set the '''Video Quality''' to medium.

<span id="finding-the-url-where-we-access-the-cameras-stream"></span>
==== 2.9 Finding the URL where we access the camera’s stream ====

Before setting up your NVR software, make sure you can view the stream using a program like VLC. Here’s how you do it:

<ol style="list-style-type: decimal;">
<li><p>'''Find the stream address''': Use <code>NMap</code> to discover all streams on port 554 (RTSP port).</p>
<pre>nmap -d --script rtsp-url-brute -p 554 192.168.5.19</pre></li>
<li><p>'''Identify streams''': Look for streams ending in <code>.sdp</code>, typically <code>stream1</code> for high quality and <code>stream2</code> for lower quality.</p></li>
<li><p>'''Modify the URL''': Adjust the RTSP URL with your username and password.</p>
<pre>rtsp://username:password@&lt;camera_ip&gt;/stream1.sdp</pre></li></ol>

<blockquote>'''Hint''': You will see the high quality &amp; the low quality stream in this list. You’ll have to mess around a bit to figure out which one is which; it should be obvious when you are viewing the high quality stream &amp; when you are viewing the low quality stream, based on the video quality.
</blockquote>
<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:lu55028jxdtp_tmp_bf8f3071.png
</gallery>

<span id="testing-streams-in-vlc"></span>
==== 2.99 Testing Streams in VLC ====

Once you’ve got the URLs, test them in '''VLC''' to ensure they work. You can click '''Media—&gt; Open Network Stream''' and then enter the URL. If you don’t have VLC… Get VLC. It is the best multi-format video player there is.

Once you have a working &amp; properly set up camera, let’s install our NVR – that stands for '''Network Video Recorder.''' This is what will monitor the video feeds coming from our cameras &amp; record it to disk for us.

<span id="step-3-installing-docker-and-setting-up-frigate-with-specific-version-0.13.2"></span>
== Step 3: Installing Docker and Setting Up Frigate with Specific Version 0.13.2 ==

Frigate is a lovely network video recorder.

Next, we’re going to clone the Frigate repository. I’m going to download Frigate, but I’m using the old version of Frigate rather than the new version. I’ll show you why once I’m done installing. The new version, in my opinion, took a well thought through user interface and destroyed it. I don’t mean minor changes; think Amber Heard doing plastic surgery on Johnny Depp. It’s that bad.

Johnny Depp would still look better after that than Frigate looked from 0.13 —&gt; 0.14. That’s what happened to Frigate from version 0.13 to 0.14. They destroyed it. You can’t even view events for more than one day at a time. It’s horrifically bad.

I’m downloading an old version, and I’ll show you the differences so you can decide for yourself. The setup routines are IDENTICAL with regards to configuring alerts in Home Assistant, etc.

This project still deserves donations, purchases, &amp; funding for how good Frigate 0.13 is, as well as thanks &amp; praise for keeping it open source so we even HAVE the option to use older versions.

<span id="install-docker"></span>
==== 3.1 Install Docker ====

<ol style="list-style-type: decimal;">
<li><p>'''Verify Existing Docker Installation:'''</p>
<p>Run the command to check if Docker is installed: <code>docker --version</code>. Make sure the version is 24.0.0 or later. If it’s an older version, remove it by using:</p>
<pre>sudo apt remove docker docker-engine docker.io containerd runc</pre></li>
<li><p>'''Install the Latest Version of Docker:'''</p>
<p>Download and install Docker using the official installation script. Run:</p>
<pre>curl -fsSL https://get.docker.com -o get-docker.sh
sudo sh get-docker.sh</pre></li></ol>

<blockquote>'''Note:''' Use the official Docker installation, not the Snap version. The Snap version is [https://www.reddit.com/r/docker/comments/shztqs/wow_docker_works_a_lot_better_when_you_dont_have/ ''horrible''] &amp; causes tons of issues. If you got tricked into installing Docker at the end of the Ubuntu server installation prompts, I am sorry, but you have to remove that, it’s garbage. Run <code>sudo snap remove docker</code> and never look back.
</blockquote>
<ol start="3" style="list-style-type: decimal;">
<li><p>'''Install Docker Compose:'''</p>
<pre>sudo apt install docker-compose-plugin -y</pre></li>
<li><p>'''Verify Docker Compose Installation:'''</p></li>
<li><p>Make sure Docker Compose version is 2.0 or higher by running:</p>
<pre>docker compose version</pre></li>
<li><p>'''Set Proper Permissions for Docker:'''</p>
<ul>
<li><p>Docker typically requires root permissions, but you can add your user to the Docker group to avoid using <code>sudo</code>. Run:</p>
<pre>sudo usermod -aG docker $USER</pre></li>
<li><p>Log out and log back in, or run:</p>
<pre>newgrp docker</pre></li></ul>
</li></ol>

<span id="install-frigate"></span>
==== 3.2 Install Frigate ====

<ol style="list-style-type: decimal;">
<li><p>'''Create a Directory for Frigate:'''</p>
<ul>
<li><p>Run the following command to create a directory to store Frigate files:</p>
<pre>mkdir -p /home/$USER/Downloads/programs
cd ~/Downloads/programs</pre></li></ul>
</li>
<li><p>'''Clone the Frigate Repository:'''</p>
<ul>
<li><p>Clone the Frigate GitHub repository by running:</p>
<pre>git clone https://github.com/blakeblackshear/frigate.git
cd frigate</pre></li></ul>
</li>
<li><p>'''Set Up Docker Compose for Frigate:'''</p>
<ul>
<li><p>Create and edit the <code>docker-compose.yml</code> file. '''Make sure it specifies Frigate version 0.13.2. New versions use a horrible user interface that is [https://www.youtube.com/watch?v=uiFLqqKkj3M&t=117s rage inducing.]''' My example file below specifies version 0.13.2 for you. You’ll need to set the container name, restart policy, image version, shared memory size, devices (e.g., USB Coral, PCIe Coral, video device for Raspberry Pi), and volumes for storing local time, config files, media, and cache. Be sure to open necessary ports (e.g., 5000, 8971, 8554, 8555).</p></li>
<li><p>*'''If any of what I said''' in the last bulletpoint after the “rage inducing” part '''confuses the hell out of you''', don’t worry: you have the easiest path there is; '''JUST COPY AND PASTE BELOW WITHOUT MESSING WITH IT!'''</p></li></ul>
</li></ol>

<pre>version: &quot;3.9&quot;
services:
  frigate:
    container_name: frigate
    privileged: true # This may not be necessary for all setups
    restart: unless-stopped
    image: ghcr.io/blakeblackshear/frigate:0.13.2 # Last good version
    shm_size: &quot;64mb&quot; # Update for your cameras based on requirements
    devices:
      - /dev/bus/usb:/dev/bus/usb # USB Coral, modify for other hardware
      - /dev/apex_0:/dev/apex_0 # PCIe Coral, modify based on your setup
      - /dev/video11:/dev/video11 # For Raspberry Pi 4B
      - /dev/dri/renderD128:/dev/dri/renderD128 # Intel hwaccel, update for your hardware
    volumes:
      - /etc/localtime:/etc/localtime:ro
      - ./config:/config
      - ./storage:/media/frigate
      - ./database:/data/db
      - type: tmpfs # Optional: Reduces SSD wear
        target: /tmp/cache
        tmpfs:
          size: 1000000000
    ports:
      - &quot;8971:8971&quot;
      - &quot;5000:5000&quot; # Internal unauthenticated access. Be careful with exposure.
      - &quot;8554:8554&quot; # RTSP feeds
      - &quot;8555:8555/tcp&quot; # WebRTC over TCP
      - &quot;8555:8555/udp&quot; # WebRTC over UDP
    environment:
      FRIGATE_RTSP_PASSWORD: &quot;password&quot;</pre>
<blockquote>'''IMPORTANT NOTE:''' This is going to record to your solid state drive for your main drive by default, which is very bad practice. The only reason it is configured this way is because we have not gotten to the zfs pool creation part of the guide, where we will create a redundant, encrypted, self-healing array of drives as a zfs pool. We want to record camera footage to large hard drives, not tiny solid state drives.

Later on in the guide, you will want to change this once ZFS is set up. The two lines of interest will be:
</blockquote>
<pre>      - ./storage:/media/frigate
      - ./database:/data/db</pre>
* This is still set to record everything to main drive: we will come back to edit this later once we have set up a ZFS pool at the end. &gt; '''DOCKER CHEAT SHEET: breaking down the <code>docker-compose.yml</code> File for Frigate'''
&gt; Every line of this <code>docker-compose.yml</code> is there for a reason. You likely have no clue what this is all for if you are reading this, so let’s go through it. &gt; &gt; '''1. <code>version: &quot;3.9&quot;</code>'''
&gt; This is the version of Docker Compose file format. Version <code>3.9</code> is compatible with new Docker setups &gt; &gt; '''2. <code>services:</code>'''
&gt; This section defines the “services” you want to run, which are containers. Here, we only have one container: <code>frigate</code>. &gt; &gt; '''3. <code>frigate:</code>'''
&gt; This is the name of the service(container). It helps you identify the container in logs or commands like <code>docker ps</code>. You can name it anything you like, but <code>frigate</code> makes sense since that’s the application we’re running. &gt; &gt; '''4. <code>container_name: frigate</code>'''
&gt; Custom name for the frigate container so it is easy to find when you type <code>docker ps -a</code> . Sometimes while debugging things that are not working you may want to enter the environment of the virtual container''(this is like sshing into your server, but into the virtual server that runs frigate)'', which you can do by typing <code>docker exec -it frigate bash</code> - but to do that you need to know which container is which! This is where using sensible names comes into play. &gt; &gt; '''5. <code>privileged: true</code>'''
&gt; Running the container in “privileged mode” allows it to access hardware devices like USB or PCIe directly. This is done because frigate can use devices you plug in(like a coral) to improve the performance of the machine learning for detecting items on camera(car, human, bird, etc) &gt; &gt; ''Warning:'' This gives the container elevated permissions, so only use it if absolutely needed (like here). &gt; &gt; '''6. <code>restart: unless-stopped</code>'''
&gt; This tells Docker to restart the container unless you stop it. If the computer reboots or the container crashes, it will turn back on automatically &gt; &gt; '''7. <code>image: ghcr.io/blakeblackshear/frigate:0.13.2</code>'''
&gt; This tells it what Docker image to use. Here, we’re pulling version <code>0.13.2</code> of Frigate from github container registry (<code>ghcr.io</code>) instead of the newest one because the user interface was tortured &amp; butchered to death with new releases. They destroyed it. It makes me sad how bad new versions are. &gt; &gt; '''8. <code>shm_size: &quot;64mb&quot;</code>'''
&gt; This sets the size of shared memory available to the container. frigate uses shared memory for hardware acceleration and video processing. frigate documentation tells you how to increase this based on how many cameras you have running. &gt; &gt; '''9. <code>devices:</code>'''
&gt; This part of the docker-compose file maps hardware devices from your host system''(the physical computer you are installing this program onto)'' into the container. Frigate needs access to specific hardware for video processing. Let’s explain each line:
&gt; &gt; - <code>/dev/bus/usb:/dev/bus/usb</code>: Maps USB devices for hardware like a USB Coral accelerator which can improve/speed up object detection &amp; take the load off of the host computer. &gt; - <code>/dev/apex_0:/dev/apex_0</code>: Maps a pci express coral thing for faster object detection.
&gt; - <code>/dev/video11:/dev/video11</code>: Maps a video input device, like a camera, for systems like Raspberry Pi.
&gt; - <code>/dev/dri/renderD128:/dev/dri/renderD128</code>: Maps Intel hardware acceleration for video encoding/decoding. &gt; &gt; '''10. <code>volumes:</code>'''
&gt; This section maps directories or volumes between the host and the container. Volumes are where we save configuration, media, and data outside the container so they continue existing even if the container is restarted/deleted/shut off. &gt; &gt; - <code>/etc/localtime:/etc/localtime:ro</code>: This maps the time of the host computer to the time of the container(“computer”) running frigate. The <code>:ro</code> means “read-only,” so the container can’t cause the host machine to time travel. Time travel is cool though. If you agree, watch the movie '''Primer''' - you won’t be disappointed. '''Triangle''' is a close second. The ending messes me up every time. &gt; - <code>./config:/config</code>: Maps the <code>config</code> directory on the host to <code>/config</code> in the container, where Frigate expects its configuration file.
&gt; - <code>./storage:/media/frigate</code>: Maps the <code>storage</code> directory on the host to <code>/media/frigate</code> in the container, where Frigate saves camera recordings.
&gt; - <code>./database:/data/db</code>: Maps the <code>database</code> directory on the host to <code>/data/db</code> in the container, where Frigate stores metadata and video analytics.
&gt; - <code>type: tmpfs</code>: Creates a temporary file system in memory. This reduces wear on SSDs by storing cache data in RAM.
&gt; - <code>target: /tmp/cache</code>: Specifies the location of the cache inside the container.
&gt; - <code>tmpfs.size: 1000000000</code>: Limits the cache size to 1 GB. &gt; &gt; '''11. <code>ports:</code>'''
&gt; This section maps network ports on the host to ports in the container. It allows you to access Frigate’s web interface and services.
&gt; - <code>&quot;8971:8971&quot;</code>: Exposes Frigate’s main web interface on port <code>8971</code>.
&gt; - <code>&quot;5000:5000&quot;</code>: Exposes an internal port for access without username/password authentication. We will fix this later using nginx &amp; an authentication setup. &gt; - <code>&quot;8554:8554&quot;</code>: Exposes Real-Time Streaming Protocol (RTSP) feeds for viewing video streams.
&gt; - <code>&quot;8555:8555/tcp&quot;</code> and <code>&quot;8555:8555/udp&quot;</code>: Expose WebRTC services over TCP and UDP, allowing low-latency streaming. &gt; &gt; '''12. <code>environment:</code>'''
&gt; This section defines environment variables, which are key-value pairs that configure the container.
&gt; &gt; - <code>FRIGATE_RTSP_PASSWORD: &quot;password&quot;</code>: Sets the password for accessing RTSP streams in Frigate. &gt; '''13. Important Warning About Default Storage'''
&gt; By default, this configuration saves camera footage (<code>./storage:/media/frigate</code>) and metadata (<code>./database:/data/db</code>) to your main drive. This is fine for testing, but long-term use will fill up and wear out your SSD. Later in the guide, you’ll learn to change these paths to a ZFS pool for redundant, self-healing storage that provides us with way more space than our operating system’d SSD.

<span id="create-frigate-configuration-file"></span>
==== 3.3 Create Frigate Configuration File ====

<ol style="list-style-type: decimal;">
<li>'''Create and Edit the <code>config.yml</code> File:'''
<ul>
<li><p>Create a <code>config/config.yml</code> file to define your cameras &amp; MQTT setup.</p></li>
<li><p>''I have provided a template below. Creating yml files is painful and very easy to mess up. So I provided a known-working file for you to start with.''</p></li>
<li><p>'''YOU WILL HAVE TO EDIT THE IP ADDRESSES, USERNAMES, AND PASSWORDS IN EACH PATH LINE TO THE URL OF YOUR ACTUAL CAMERA. YOUR CAMERAS WILL ALSO HAVE DIFFERENT URLS THAN MINE. I DID MOST OF THE WORK FOR YOU, BUT DON’T BE SO LAZY THAT YOU DON’T EVEN CHANGE THE CAMERA IPs &amp; USERNAMES &amp; PASSWORDS TO YOURS!'''</p></li>
<li><p>To find the RTSP URLs of your camera, you can install <code>nmap</code> on Ubuntu with:</p>
<pre>sudo apt install nmap -y</pre></li>
<li><p>Then you go to your terminal and type the following, replacing the IP address of <code>192.168.3.120</code> with the IP address of your camera:</p>
<pre>sudo nmap --script rtsp-url-brute -p 554 192.168.5.19
sudo nmap --script rtsp-url-brute -p 8554 192.168.5.19</pre></li>
<li><p>You will receive a list of stream URLs. Let’s say one of them is <code>&quot;rtsp://192.168.5.19/Streaming/Channels/101&quot;</code>.</p></li></ul>
</li></ol>

* You need to add your username &amp; password here. So <code>rtsp://192.168.5.19/Streaming/Channels/101</code> will become <code>rtsp://username:password@192.168.5.19/Streaming/Channels/101</code>.
* Test that this works in a video player like VLC. In VLC, go '''Media''' → '''Open Network Stream''' → '''Network URL''' → enter the URL → click '''Play'''.
* If it works, it can be entered into the <code>path</code> line and replace my URLs in the config file below.
* The first four lines are going to be for MQTT, which sends messages to Home Assistant so that Home Assistant can send alerts to your phone when someone tries to steal your catalytic converter.

<pre>mqtt:
  host: homeassistant.home.arpa  
  port: 1883
  user: louis
  password: passwordman

cameras:
  front_door_closeup:
    ffmpeg:
      inputs:
        - path: rtsp://CAMERAUSERNAMEGOESHERE:CAMERAPASSWORDGOESHERE@192.168.3.101:554/Streaming/Channels/101
          roles:
            - record
        - path: rtsp://CAMERAUSERNAMEGOESHERE:CAMERAPASSWORDGOESHERE@192.168.3.101:554/Streaming/Channels/102
          roles:
            - detect
      output_args:
        record: -f segment -segment_time 60 -segment_format mp4 -reset_timestamps 1 -strftime 1 -c copy
    detect:
      width: 640
      height: 360
      fps: 20
    objects:
      track:
        - person
        - car
        - motorcycle
        - bird
        - cat
        - dog
        - horse
        - sheep
        - cow
        - bear
        - zebra
        - giraffe
        - elephant
        - mouse
      filters:
        person:
          mask: 570,299,545,0
        cat:
          min_score: 0.01
          threshold: 0.02
        dog:
          min_score: 0.01
          threshold: 0.02
        bird:
          min_score: 0.01
          threshold: 0.02
    motion:
      mask:
        - 473,0,21,156,53,317,140,312
    record:
      enabled: true
      events:
        pre_capture: 5
        post_capture: 5
        objects:
          - person
          - car
          - motorcycle
          - bird
          - cat
          - dog
          - horse
          - sheep
          - cow
          - bear
          - zebra
          - giraffe
          - elephant
          - mouse

  driveway:
    ffmpeg:
      inputs:
        - path: rtsp://CAMERAUSERNAMEGOESHERE:CAMERAPASSWORDGOESHERE@192.168.3.102:554/Streaming/Channels/101
          roles:
            - record
        - path: rtsp://CAMERAUSERNAMEGOESHERE:CAMERAPASSWORDGOESHERE@192.168.3.102:554/Streaming/Channels/102
          roles:
            - detect
      output_args:
        record: -f segment -segment_time 60 -segment_format mp4 -reset_timestamps 1 -strftime 1 -c copy
    detect:
      width: 640
      height: 360
      fps: 20
    objects:
      track:
        - person
        - car
        - motorcycle
        - bird
        - cat
        - dog
        - horse
        - sheep
        - cow
        - bear
        - zebra
        - giraffe
        - elephant
        - mouse
      filters:
        car:
          min_score: 0.01
          threshold: 0.03
        cat:
          min_score: 0.01
          threshold: 0.02
        dog:
          min_score: 0.01
          threshold: 0.02
        bird:
          min_score: 0.01
          threshold: 0.02
    record:
      enabled: true
      events:
        pre_capture: 5
        post_capture: 5
        objects:
          - person
          - car
          - motorcycle
          - bird
          - cat
          - dog
          - horse
          - sheep
          - cow
          - bear
          - zebra
          - giraffe
          - elephant
          - mouse

  side_door_closeup:
    ffmpeg:
      inputs:
        - path: rtsp://CAMERAUSERNAMEGOESHERE:CAMERAPASSWORDGOESHERE@192.168.3.104:554/Streaming/Channels/101
          roles:
            - record
        - path: rtsp://CAMERAUSERNAMEGOESHERE:CAMERAPASSWORDGOESHERE@192.168.3.104:554/Streaming/Channels/102
          roles:
            - detect
      output_args:
        record: -f segment -segment_time 60 -segment_format mp4 -reset_timestamps 1 -strftime 1 -c copy
    detect:
      width: 640
      height: 360
      fps: 20
    objects:
      track:
        - person
        - bird
        - cat
        - dog
        - horse
        - sheep
        - cow
        - bear
        - zebra
        - giraffe
        - elephant
        - mouse
      filters:
        car:
          min_score: 0.01
          threshold: 0.03
        cat:
          min_score: 0.01
          threshold: 0.02
        dog:
          min_score: 0.01
          threshold: 0.02
        bird:
          min_score: 0.70
          threshold: 0.75
    record:
      enabled: true
      events:
        pre_capture: 5
        post_capture: 5
        objects:
          - person
          - car
          - bird
          - cat
          - dog
          - horse
          - sheep
          - cow
          - bear
          - zebra
          - giraffe
          - elephant
          - mouse

  back_door_closeup:
    ffmpeg:
      inputs:
        - path: rtsp://CAMERAUSERNAMEGOESHERE:CAMERAPASSWORDGOESHERE@192.168.3.103:554/Streaming/Channels/101
          roles:
            - record
        - path: rtsp://CAMERAUSERNAMEGOESHERE:CAMERAPASSWORDGOESHERE@192.168.3.103:554/Streaming/Channels/102
          roles:
            - detect
      output_args:
        record: -f segment -segment_time 60 -segment_format mp4 -reset_timestamps 1 -strftime 1 -c copy
    detect:
      width: 640
      height: 360
      fps: 20
    objects:
      track:
        - person
        - car
        - bird
        - cat
        - dog
        - horse
        - sheep
        - cow
        - bear
        - zebra
        - giraffe
        - elephant
        - mouse
      filters:
        car:
          min_score: 0.75
          threshold: 0.75
        cat:
          min_score: 0.01
          threshold: 0.02
        dog:
          min_score: 0.01
          threshold: 0.02
        bird:
          min_score: 0.01
          threshold: 0.02
    record:
      enabled: true
      events:
        pre_capture: 5
        post_capture: 5
        objects:
          - person
          - car
          - bird
          - cat
          - dog
          - horse
          - sheep
          - cow
          - bear
          - zebra
          - giraffe
          - elephant
          - mouse

  front_porch_wide_angle:
    ffmpeg:
      inputs:
        - path: rtsp://CAMERAUSERNAMEGOESHERE:CAMERAPASSWORDGOESHERE@192.168.3.106:554/Streaming/Channels/101
          roles:
            - record
        - path: rtsp://CAMERAUSERNAMEGOESHERE:CAMERAPASSWORDGOESHERE@192.168.3.106:554/Streaming/Channels/102
          roles:
            - detect
      output_args:
        record: -f segment -segment_time 60 -segment_format mp4 -reset_timestamps 1 -strftime 1 -c copy
    detect:
      width: 640
      height: 360
      fps: 20
    objects:
      track:
        - person
        - car
        - motorcycle
        - bird
        - cat
        - dog
        - horse
        - sheep
        - cow
        - bear
        - zebra
        - giraffe
        - elephant
        - mouse
      filters:
        person:
          min_score: 0.8
          threshold: 0.8
        car:
          min_score: 0.6
          threshold: 0.7
        cat:
          min_score: 0.01
          threshold: 0.02
        dog:
          min_score: 0.01
          threshold: 0.02
        bird:
          min_score: 0.6
          threshold: 0.65
    record:
      enabled: true
      events:
        pre_capture: 5
        post_capture: 5
        objects:
          - person
          - car
          - motorcycle
          - bird
          - cat
          - dog
          - horse
          - sheep
          - cow
          - bear
          - zebra
          - giraffe
          - elephant
          - mouse

  fishcam:
    ffmpeg:
      inputs:
        - path: rtsp://louis:passwordroflcopter@192.168.3.120:554/stream1
          roles:
            - record
        - path: rtsp://louis:passwordroflcopter@192.168.3.120:554/stream1
          roles:
            - detect
      output_args:
        record: -f segment -segment_time 60 -segment_format mp4 -reset_timestamps 1 -strftime 1 -c copy
    detect:
      width: 640
      height: 360
      fps: 20
    objects:
      track:
        - person
      filters:
        person:
          min_score: 0.3
          threshold: 0.3
    record:
      enabled: true
      events:
        pre_capture: 15
        post_capture: 15
        objects:
          - fish

database:
  path: /data/db/frigate.db
#version: 0.14</pre>
<blockquote>'''Note:''' For each camera, configure the RTSP inputs for recording and detection streams. Define output arguments, detection settings (e.g., width, height, fps), and tracked objects (e.g., person, car, bird, dog). You can set filters for specific objects, mask areas for motion detection, and enable event recording with pre-capture and post-capture times. Repeat for additional cameras as needed.
</blockquote>
<span id="running-frigate"></span>
==== 3.4 Running Frigate ====

# '''Start Frigate:'''
#* Start Frigate by running: <code>docker compose up -d</code>.
# '''Access the Frigate Web Interface:'''
#* Open your web browser and navigate to <code>http://192.168.5.2:5000</code>.
# '''Configure Additional Settings:'''
#* Edit the <code>config.yml</code> file as needed to add or modify cameras, object tracking settings, or motion detection masks.
# '''Note on Storage:'''
#* It’s recommended to use a separate storage device for Frigate’s media to avoid unnecessary wear on your primary SSD. We’ll go into detail about setting up ZFS pools &amp; external storage later.

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:lu55028jxdtp_tmp_54097ca6.png
</gallery>

<span id="enjoy-frigate"></span>
==== 3.5 Enjoy Frigate! ====

You have the best NVR software there is, and no cancerous hideous modern UI. Enjoy!

<span id="step-4-make-sure-it-all-works."></span>
== Step 4: Make sure it all works. ==

There’s nothing worse than someone kidnapping your kid or killing your dog &amp; not being able to see who did it because you set your threshold too low in a yaml file. Extensively test everything. Assume it won’t work later, because often with camera systems, it doesn’t.

<span id="step-5-get-instant-camera-alerts-on-your-phone"></span>
== Step 5: Get Instant Camera Alerts On Your Phone ==

Now you have a camera you can see when you log into it, but don’t you want to get an alert if some weirdo is walking through your backyard? Home Assistant and Frigate can talk to each other to make this happen.

Home Assistant needs two things:

* To receive communication from Frigate
* A client and a broker that understand that communication.

We are going to go over how to set all of this up – and use a handy extension that allows us to avoid miserable YAML files for setting this all up, that is simple, point, and click.

<span id="switch-gears-go-back-to-home-assistant"></span>
==== 5.1 Switch gears &amp; go back to Home Assistant ====

# Open web browser
# Go to http://192.168.1.7:8123 or http://homeassistant.home.arpa:8123

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:lu55028jxdtp_tmp_73352a27.png
File:lu55028jxdtp_tmp_fc4eb41f.png
File:lu55028jxdtp_tmp_12b8030b.png
File:lu55028jxdtp_tmp_deda9e79.png
File:lu55028jxdtp_tmp_598cb682.png
File:lu55028jxdtp_tmp_68982d5e.png
File:lu55028jxdtp_tmp_d991780a.png
File:lu55028jxdtp_tmp_b201bbe8.png
File:lu55028jxdtp_tmp_272b62c8.png
File:lu55028jxdtp_tmp_16db9dd9.png
File:lu55028jxdtp_tmp_50c0fdd2.png
File:lu55028jxdtp_tmp_f69d8e0e.png
File:lu55028jxdtp_tmp_88fe4866.png
File:lu55028jxdtp_tmp_ee1dcb43.png
File:lu55028jxdtp_tmp_904be7b9.png
File:lu55028jxdtp_tmp_21da8192.png
File:lu55028jxdtp_tmp_55120826.png
File:lu55028jxdtp_tmp_7c2ff154.png
File:lu55028jxdtp_tmp_b0c46153.png
File:lu55028jxdtp_tmp_44a26f8c.png
File:lu55028jxdtp_tmp_e003f8b9.png
File:lu55028jxdtp_tmp_78a73239.png
File:lu55028jxdtp_tmp_aff05f0f.png
File:lu55028jxdtp_tmp_cf418dfc.png
File:lu55028jxdtp_tmp_584df2f7.png
File:lu55028jxdtp_tmp_e3b61efd.png
File:lu55028jxdtp_tmp_8749f406.png
File:lu55028jxdtp_tmp_99985c72.png
</gallery>

<span id="download-and-install-hacs"></span>
==== 5.2 Download and Install HACS ====

# '''Download HACS (Home Assistant Community Store):'''
#* Go to [https://www.hacs.xyz/docs/use/download/download/ HACS → Download] on their website.
#* Click onto the '''OS/supervised''' version, as that’s the version of Home Assistant we have installed.
# '''Open the HACS Add-on Repository:'''
#* Click the link provided to add the HACS repository to your Home Assistant instance. It’ll ask you to '''Add missing'''.
# '''Enter Home Assistant URL:'''
#* It will ask for your Home Assistant link.
#* By default, Home Assistant may attempt to use <code>homeassistant.local:8123</code>, which will fail.
#* If you are following this guide’s setup, use one of the following URLs:
#** Local Domain: <code>http://homeassistant.home.arpa:8123</code>
#** Direct IP: <code>http://192.168.5.4:8123</code>
#* Replace these with your actual Home Assistant domain or IP address if different.
# '''Install HACS:'''
#* Follow the prompts to install HACS in Home Assistant.
#* ''BE PATIENT!'' Click on the LOGS tab and wait for it to be '''''DONE!!!''''' before you try to start adding things, or nothing will work.
# '''Restart Home Assistant:'''
#* After installation, restart your Home Assistant instance for the changes to take effect.
#* Go to settings → system → power button icon in the upper right-hand corner, click the power button, and click “restart home assistant.” '''''DO NOT DO THIS UNTIL THE LOGS TAB FOR HACS SAYS EVERYTHING IS DONE'''''
# '''Clear your browser cache, cookies, etc.'''
# '''Log back into Home Assistant.'''
# '''Go to Settings → Devices &amp; Services → Add Integration &amp; Search for HACS'''
#* If it doesn’t show up, do not pass go, do not collect $200 – re-follow the instructions [https://www.hacs.xyz/docs/use/download/download/ here] and [https://www.hacs.xyz/docs/use/configuration/basic/#setting-up-the-hacs-integration here]. Clear your browser cache/cookies, choose the option to reboot Home Assistant rather than restart Home Assistant when you go to settings → system → power button icon in the upper right-hand corner, clear cache/cookies in the browser, go to settings → addons → get HACS → CLICK START.
# '''Go to logs'''

<ul>
<li><p>''Wait! Don’t be impatient!'' Wait for it to be done. You will see the following at the end of the log when it is done:</p>
<pre>INFO: Installation complete.
INFO: Remember to restart Home Assistant before you configure it.
s6-rc: info: service legacy-services: stopping
s6-rc: info: service legacy-services successfully stopped
s6-rc: info: service legacy-cont-init: stopping
s6-rc: info: service legacy-cont-init successfully stopped
s6-rc: info: service fix-attrs: stopping
s6-rc: info: service fix-attrs successfully stopped
s6-rc: info: service s6rc-oneshot-runner: stopping
s6-rc: info: service s6rc-oneshot-runner successfully stopped</pre></li></ul>

<ol start="10" style="list-style-type: decimal;">
<li><p>'''Add Integration Properly:'''</p>
<ul>
<li><p>Go to '''Settings –&gt; Devices → Devices &amp; Integrations → Add Integration''' &amp; search for HACS.</p></li>
<li><p>Check the boxes.</p></li>
<li><p>Click submit.</p></li>
<li><p>It will ask you to open a link to log into GitHub, and insert a code. Click it.</p></li>
<li><p>Go to GitHub. If you lack an account, make one. If you have a GitHub account, log in.</p></li>
<li><p>Enter code.</p></li>
<li><p>Authorize HACS.</p></li>
<li><p>Add HACS to an “area.”</p></li>
<li><p>Click finish.</p></li>
<li><p>Next step!</p></li></ul>
</li></ol>

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:lu55028jxdtp_tmp_4a627a28.png
File:lu55028jxdtp_tmp_6e463f1.png
File:lu55028jxdtp_tmp_25f1321.png
File:lu55028jxdtp_tmp_bd48c948.png
File:lu55028jxdtp_tmp_7d0028c9.png
File:lu55028jxdtp_tmp_136e049.png
File:lu55028jxdtp_tmp_ce8689b1.png
File:lu55028jxdtp_tmp_c6badc81.png
File:lu55028jxdtp_tmp_d4c54e09.png
File:lu55028jxdtp_tmp_69f853e4.png
File:lu55028jxdtp_tmp_c6c7c21e.png
File:lu55028jxdtp_tmp_d892307f.png
File:lu55028jxdtp_tmp_e2f2eaf6.png
File:lu55028jxdtp_tmp_b15f4ff6.png
File:lu55028jxdtp_tmp_f2ef1560.png
File:lu55028jxdtp_tmp_408094f6.png
</gallery>

<span id="add-frigate-add-ons-to-home-assistant"></span>
==== 5.3 Add Frigate Add-ons to Home Assistant ====

# Visit [http://homeassistant.home.arpa:8123/hacs/repository/311536795 '''Frigate Home Assistant Add-ons page''']
# '''Log back into Home Assistant when it prompts you to.'''
# '''Add Frigate Repository:'''
#* Click the bright blue '''“Add-on repository to my Home Assistant”''' button.
# '''Download and Install Frigate:'''
#* You’ll see two buttons. One is a blue button that says “Open with Home Assistant Store,” and the other is for downloading the add-on.
#* '''Important:''' The blue button in the middle refreshes the page without installing anything.
#* To download and install Frigate, make sure to click the Download button at the bottom.
# '''Access Home Assistant Again:'''
#* You’ll be prompted again to enter your Home Assistant domain with <code>:8123</code>.
#* Remember, the default URL <code>homeassistant.local:8123</code> won’t work. HomeAssistant assumes you’re using a standard router where the domain is <code>.local</code> - but with pfsense, it is <code>.home.arpa</code> Use one of the following:
#** '''Local Domain:''' [http://homeassistant.home.arpa:8123/ http://homeassistant.home.arpa:8123]
#** '''Direct IP:''' [http://192.168.5.4:8123/ http://192.168.5.4:8123]
# Click '''“Download”''' in the lower left corner.
# Continue with installing, wait for it to install — it should be quick.
# Go to '''Home Assistant Settings''' in the lower left corner.
# It will say '''“1 repair, restart required”''' with the little Frigate logo at the top, or just '''restart required''' at the top.
# Click this, follow prompts, and restart Home Assistant.

<span id="add-frigate-integration"></span>
==== 5.4 Add Frigate Integration ====

<ol style="list-style-type: decimal;">
<li><p>'''Add Frigate integration to Home Assistant'''</p>
<ol style="list-style-type: lower-alpha;">
<li><p>Go to '''Settings''' in the Home Assistant menu.</p></li>
<li><p>Navigate to '''Devices &amp; Integrations'''.</p></li>
<li><p>Click Add Integration, and search for Frigate in the list. Follow the prompts to add it.</p></li></ol>
</li>
<li><p>'''Enter Frigate URL:'''</p>
<ol style="list-style-type: lower-alpha;">
<li>The URL will be the IP address you chose for the server you installed Frigate on, or its hostname: in my case <code>http://192.168.5.2:5000</code>, OR [http://happycloud.home.arpa:5000/ http://happycloud.home.arpa:5000] with the examples I have provided.</li></ol>
</li>
<li><p>Once Frigate is integrated, you’ll be asked to assign cameras to specific areas within Home Assistant. Select the appropriate areas for your cameras.</p></li></ol>

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:lu55028jxdtp_tmp_11421791.png
File:lu55028jxdtp_tmp_b46921b6.png
File:lu55028jxdtp_tmp_20eefd9e.png
File:lu55028jxdtp_tmp_240785dd.png
File:lu55028jxdtp_tmp_21cd7be8.png
File:lu55028jxdtp_tmp_63ccf803.png
File:lu55028jxdtp_tmp_6c42c4c.png
File:lu55028jxdtp_tmp_a1ab324e.png
File:lu55028jxdtp_tmp_37888578.png
File:lu55028jxdtp_tmp_4d8907f9.png
File:lu55028jxdtp_tmp_61ecad83.png
File:lu55028jxdtp_tmp_20c1c405.png
</gallery>

<span id="configure-mosquito-broker-mqtt-in-that-order"></span>
==== 5.5: Configure Mosquito Broker &amp; MQTT (in that order) ====

# '''Check if MQTT Broker (Mosquitto) is Installed:''' Go to '''Settings &gt; Add-ons''' and find the blue '''add-on Store button''' at the bottom right.
# '''Look for Mosquitto Broker.'''
# '''Click Install.'''
# Once installed, start the add-on and make sure Start on Boot is enabled, and hit start.
# '''Configure MQTT Broker in Home Assistant:'''
#* Go to '''Settings &gt; Devices &amp; Services &gt; Add Integration'''.
#* Search for '''MQTT''' and select it. Go into MQTT by clicking it and add it.
# '''Autoconfigure Prompt:'''
#* It should prompt you to autoconfigure it with the mosquito broker you just installed.
#* Remember the order – install mosquito broker from addons FIRST, THEN install MQTT from '''Settings &gt; Devices &amp; Services &gt; Add Integration''', or MQTT may not auto-configure itself the same way.
#** Broker: <code>core-mosquitto</code> (since Mosquitto is running on Home Assistant OS). This will auto configure by default.
#** Don’t worry if the MQTT thing has no working configure buttons, those are as optional as the JTAG connector on a MacBook motherboard.
#** Port: 1883 (default MQTT port). This will auto configure by default.
#** Username and Password: Mosquitto broker allows Home Assistant users to log in so you don’t have to worry about this. When we enter this information into Frigate, we will be using the username &amp; password we use to log into home assistant.

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:lu55028jxdtp_tmp_ca1b97c5.png
File:lu55028jxdtp_tmp_b6564f8b.png
File:lu55028jxdtp_tmp_c414c9.png
File:lu55028jxdtp_tmp_21dbe66b.png
File:lu55028jxdtp_tmp_1008176c.png
File:lu55028jxdtp_tmp_a1f44605.png
</gallery>

<span id="set-up-frigate-mobile-app-notifications"></span>
==== 5.6 Set Up Frigate Mobile App Notifications ====

* '''Download Notification Blueprint:'''
** Go to the [https://community.home-assistant.io/t/frigate-mobile-app-notifications-2-0/559732 Frigate Mobile App Notifications] 2.0 page.
** Follow the instructions on this page to download the notification blueprint into your Home Assistant.
* '''You need this unless you want to be in hell writing YAML files yourself. You don’t want to do that, right? I thought so.'''

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:lu55028jxdtp_tmp_cf0e970f.png
File:lu55028jxdtp_tmp_b21d7dac.png
File:lu55028jxdtp_tmp_34426308.png
File:lu55028jxdtp_tmp_3ddaeebf.png
File:lu55028jxdtp_tmp_240dc338.png
File:lu55028jxdtp_tmp_bed46eef.png
File:lu55028jxdtp_tmp_ab6084f9.png
File:lu55028jxdtp_tmp_837df932.png
File:lu55028jxdtp_tmp_6e9bae11.png
File:lu55028jxdtp_tmp_d45994a6.png
File:lu55028jxdtp_tmp_a8dc04b5.png
File:lu55028jxdtp_tmp_bd7663b7.png
File:lu55028jxdtp_tmp_e21b65cd.png
File:lu55028jxdtp_tmp_cf25be7a.png
File:lu55028jxdtp_tmp_7ad3f32e.png
</gallery>

<span id="configure-automations-for-camera-and-notifications"></span>
==== 5.7 Configure Automations for Camera and Notifications ====

<ol style="list-style-type: decimal;">
<li><p>'''Access Automation Editor:'''</p>
<ul>
<li>Go to your automation editor at: <code>http://192.168.5.4:8123/config/automation/dashboard</code></li></ul>
</li>
<li><p>'''Use Frigate Notifications Blueprint:'''</p>
<ul>
<li>Click '''“Blueprints”''' at the top right.</li>
<li>Click '''“Frigate Notifications”''' which is what you want.</li></ul>
</li>
<li><p>'''Configure Automation:'''</p>
<ul>
<li><p>Here you scroll down to choose your camera, and your mobile device, the name of the automation, etc.</p></li>
<li><p>Most important thing to get right is the name of the camera &amp; the mobile device, everything else you can customize and it’s not for me to tell you how to.</p>
<blockquote><p>'''NOTE:''' If your mobile device does not show up, log into Home Assistant on your phone and add it as a device to Home Assistant. It will prompt you to do this by default when you first set up the app. Then go back here and redo this step (you will have to close out of the window you just opened after clicking '''Blueprints → Frigate Notifications''' &amp; reclick it so the dialog box for your phone will show your phone)</p></blockquote></li></ul>
</li>
<li><p>'''Make sure MQTT is set up in the <code>frigate config.yml</code> file:'''</p>
<ul>
<li>Make sure in Frigate’s '''Config''' menu, in the <code>config.yml</code> file, MQTT is set up as follows, with the username &amp; password matching your homeassistant login, and your host matching the IP address of the home assistant server:</li></ul>
</li></ol>

<pre>mqtt:
  host: homeassistant.home.arpa
  port: 1883
  user: louis
  password: passwordman</pre>
<ol start="6" style="list-style-type: decimal;">
<li>'''Enjoy Your New Frigate Integration with Home Assistant!'''</li></ol>

<span id="step-6-making-frigate-secure"></span>
== Step 6: Making Frigate Secure ==

<blockquote>'''NOTE:''' (if the complexities of docker networking confuse you, skip ahead to “steps”)
</blockquote>
Newer frigate has username/password authentication, but it is so useless you will not want to ever log into it. That isn’t helpful.

Older frigate has no authentication, so anyone who goes to <code>http://192.168.5.2:5000</code> on your local network has admin access to everything. They can stop recording, delete recordings, have your setup record [https://en.wikipedia.org/wiki/Goatse.cx goatse], etc. '''VERY BAD'''.

Further complicating things, our Frigate plugin on Home Assistant, at <code>192.168.5.4</code>, needs to communicate with <code>192.168.5.2</code> in order to grab Frigate’s camera setup, on port 5000 – WITHOUT authentication. The communication to grab the camera setup is separate from the mqtt traffic. :( This makes it difficult to secure versions of Frigate that have a functioning UI.

We can set up <code>nginx</code> as a reverse proxy – this directs all traffic that is received on port 80 &amp; 443 to <code>https://</code> traffic that directs to Frigate on port 5000. We can add username/password authentication using <code>nginx</code> here, so that people need a password to view it. Then, we can block port 5000 by binding Frigate to only work on localhost.

But this means that Home Assistant won’t be able to connect to it – since it’s running on another machine. '''F&amp;^!'''

* Plan to set up username/password authentication for Frigate:
** Use <code>iptables</code> to allow all traffic to port 5000 from <code>127.0.0.1</code> (localhost, the computer running Frigate), so that <code>nginx</code> can connect to Frigate.
** Allow all traffic from <code>192.168.5.4</code>, our Home Assistant virtual machine, to connect to port 5000 Frigate.
** Block EVERYTHING ELSE on port 5000.
** Set up <code>nginx</code> as a webserver on port 443 with https &amp; ssl.
** Tell <code>nginx</code> anyone accessing the <code>nginx</code> webserver they need to submit a username &amp; password to get in.
** Tell <code>nginx</code> to show anyone who enters that user/pass when showing up on port 443 to be able to see Frigate on port 5000.

'''TL;DR'''

* We’re telling everyone who wants to view the cameras they have to enter a username &amp; password.
* This allows you to view your cameras just fine.
* This tells anyone who tries to get into your system without a password to gargle your balls.
* This allows homeassistant to connect without being blocked.

We have to do this on the machine itself, since people on our LAN are not going to have to talk to the router in order to log into Frigate, since they are on the same network. These rules will be added on <code>192.168.5.2</code>, aka <code>happycloud.home.arpa</code>, our machine that is running Frigate.

<span id="making-iptables-rules"></span>
==== 6.1 Making iptables rules ====

Allow established connections (makes https more stable, [https://www.reddit.com/r/radiohead/comments/ovvkrg/understanding_ok_computers_fitter_happier/ fitter, happier, more productive]. Not eating too much)

<pre>sudo iptables -A INPUT -m state --state ESTABLISHED,RELATED -j ACCEPT</pre>
<blockquote>Allow localhost access to port 5000:
</blockquote>
<pre>sudo iptables -A INPUT -i lo -p tcp --dport 5000 -j ACCEPT</pre>
<blockquote>Allow Home Assistant access to port 5000
</blockquote>
<pre>sudo iptables -A INPUT -s 192.168.5.4 -p tcp --dport 5000 -j ACCEPT</pre>
<blockquote>Block all other access to port 5000
</blockquote>
<pre>sudo iptables -A INPUT -p tcp --dport 5000 -j DROP</pre>
<blockquote>Make sure Docker respects these rules
</blockquote>
<pre>sudo iptables -I DOCKER-USER -j RETURN</pre>
<blockquote>Install the <code>iptables-persistent</code> package:
</blockquote>
<pre>sudo apt install iptables-persistent</pre>
# During installation, you’ll be asked if you want to save the current iptables rules. Choose Yes.
# If you’re not prompted, you can manually save the rules by running: <code>sudo netfilter-persistent save</code>
# YOU NEED TO INSTALL <code>IPTABLES-PERSISTENT</code> AND TELL IT TO SAVE YOUR RULES OR ELSE YOU HAVE TO RUN THIS EVERY TIME YOU BOOT!

<span id="installing-nginx"></span>
==== 6.2 Installing nginx ====

Next up, it’s time to install nginx &amp; everything necessary for us to have it ask for a username and a password to log in.

<ol style="list-style-type: decimal;">
<li><p>'''Install Nginx:'''</p>
<pre>sudo apt install nginx</pre></li>
<li><p>'''Run the following commands:'''</p>
<pre>sudo apt update
sudo apt install nginx -y</pre></li>
<li><p>'''Create a Self-Signed SSL Certificate'''
Generate the certificate:</p>
<pre>sudo openssl req -x509 -nodes -days 365 -newkey rsa:2048 -keyout /etc/ssl/private/nginx-selfsigned.key -out /etc/ssl/certs/nginx-selfsigned.crt</pre></li></ol>

<blockquote>'''Note:''' For the Common Name (CN), use your local domain (e.g., happycloud.home.arpa).
</blockquote>
<ol start="4" style="list-style-type: decimal;">
<li><p>'''Create Strong Diffie-Hellman Group''', makes security and https better, because we totally need more security on a LAN connection nobody else will be able to connect to besides your kid who’s trying to troll you with</p>
<pre>sudo openssl dhparam -out /etc/ssl/certs/dhparam.pem 2048</pre></li>
<li><p>'''Create Password File for Basic Auth'''
Install apache2-utils and create the password file:</p>
<pre>sudo apt install apache2-utils
sudo htpasswd -c /etc/nginx/.htpasswd your_username</pre></li>
<li><p>Replace your_username with your desired username.</p></li></ol>

<span id="configure-nginx"></span>
==== 6.3 Configure Nginx ====

Create a new Nginx configuration file:

<pre>sudo nano /etc/nginx/sites-available/frigate</pre>
If this directory does not exist, you might be using a newer version of nginx, which places configuration files in <code>/etc/nginx/conf.d</code> instead. Running <code>nginx -v</code> will tell you whether you are using an older version that defaults to [https://www.reddit.com/r/nginx/comments/re8ksm/why_is_sitesenabled_and_sitesavailable_deprecated/ <code>/etc/nginx/sites-available</code> and <code>/etc/nginx/sites-enabled</code>] or a newer version that uses <code>/etc/nginx/conf.d/</code> in that case:

<pre>sudo nano /etc/nginx/sites-available/frigate</pre>
Add the following configuration: remember to replace '''“happycloud.home.arpa”''' as well as '''“192.168.5.2”''' with the hostname &amp; IP address of YOUR server!

<pre>server {
    listen 80;
    server_name happycloud.home.arpa 192.168.5.2;
    return 301 https://$host$request_uri;
}
server {
    listen 443 ssl;
    server_name happycloud.home.arpa 192.168.5.2;
    ssl_certificate /etc/ssl/certs/nginx-selfsigned.crt;
    ssl_certificate_key /etc/ssl/private/nginx-selfsigned.key;
    ssl_dhparam /etc/ssl/certs/dhparam.pem;
    ssl_session_timeout 10m;
    ssl_session_cache shared:SSL:10m;
    ssl_session_tickets off;
    auth_basic &quot;Restricted Access&quot;;
    auth_basic_user_file /etc/nginx/.htpasswd;
    location / {
        proxy_pass http://127.0.0.1:5000;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
    }
    location /ws {
        proxy_pass http://127.0.0.1:5000;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection &quot;upgrade&quot;;
        proxy_set_header Host $host;
    }
}</pre>
<blockquote>'''NOTE''': Many open source projects suggest using nginx as a reverse proxy. They are kind &amp; cordial enough to provide their own configuration files for you so you don’t have to write everything above &amp; configure it yourself.

While well meaning, many of them set the cipher(security thingie) manually, a throwback tot he days when nginx used to default to insecure ciphers. So you may see old docs by developers that MEANT WELL to provide you a helping hand with stuff like this in their nginx configuration files:
</blockquote>
<pre>    ssl_protocols TLSv1.2 TLSv1.3;
    ssl_prefer_server_ciphers on;
    ssl_ciphers ECDHE-RSA-AES256-GCM-SHA512:DHE-RSA-AES256-GCM-SHA512:ECDHE-RSA-AES256-SHA384:DHE-RSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:DHE-RSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384;</pre>
<blockquote>This is bad. Remove things like this as long as you are using a modern version of nginx. These change often and if you are manually setting it, that is not a great thing to be doing. Also consider politely''(POLITELY)'' mentioning to the devs who had that in there that this isn’t necessary anymore since nginx no longer defaults to insecure ciphers.
</blockquote>
<span id="enable-the-nginx-configuration"></span>
==== 6.4 Enable the Nginx Configuration ====

Enable the configuration and reload Nginx. The commands below do the following:

<code>sudo ln -s /etc/nginx/sites-available/frigate /etc/nginx/sites-enabled/</code> This takes our configuration file out of the “chamber” (sites-available) and into the breech (sites-enabled). Your configuration file you place in sites-available will not work unless it is in sites-enabled. <code>ln -s</code> creates a symlink, similar to how a shortcut works in Windows. <code>nginx -t</code> checks our configuration file for errors. <code>sudo systemctl reload nginx</code> allows nginx to load a new configuration file ''without'' shutting down.

<pre>sudo ln -s /etc/nginx/sites-available/frigate /etc/nginx/sites-enabled/ 
sudo nginx -t # This checks if config is bad &amp; tells us what we did wrong
sudo systemctl reload nginx</pre>
<span id="test-frigate-does-it-require-userpass"></span>
==== 6.5 Test Frigate; does it require user/pass? ====

* Log into <code>http://192.168.5.2:5000</code> from other computers on your LAN. If it doesn’t work, you did good.
* Your nephew can no longer replace your cameras with goatse.

<span id="make-sure-home-assistant-still-connects-to-frigate."></span>
==== 6.6 Make sure Home Assistant still connects to Frigate. ====

# Go over to Home Assistant at <code>http://192.168.5.4:8123</code> or <code>http://homeassistant.home.arpa</code>
# Go to '''Settings → Devices &amp; Integrations → Frigate'''
# Click '''“ADD DEVICE”''' blue button on bottom right
# Enter the IP or hostname, along with port, of the machine running Frigate on port 5000 like such: <code>http://192.168.5.2:5000</code> or <code>http://happycloud.home.arpa:5000</code>
# Click '''SUBMIT'''.
# If Home Assistant’s Frigate plugin can find your cameras by connecting to Frigate on port 5000, but no other computer on your LAN can, YOU DID GOOD.
# Go to http://192.168.5.2 – it should auto-redirect you to SSL https://192.168.5.2 &amp; then ask for username &amp; password.
# Enter your username &amp; password.
# If you are now in Frigate, you done good.

<span id="replacing-google-drive-photos-docs-sheets-keep"></span>
= Replacing Google Drive, Photos, Docs, Sheets, &amp; Keep =

Next up, we’ll be setting up a complete app suite so those of you used to iCloud for photos, Google Docs for online office, backup, etc., don’t feel like you’re making big sacrifices. The programs we’ll be installing are as follows:

# Immich, to replace Google Photos/iCloud Photos
# Onlyoffice, to replace Google Docs &amp; Google Sheets
# Syncthing, to replace iCloud &amp; Google Drive
# Samba, to allow easy access in any file explorer in any operating system to users connected via VPN
# Nextcloud Notes for a Google Keep-like notes system.

<span id="step-1-making-a-new-virtual-machine"></span>
== Step 1: Making a new virtual machine ==

We are going to create a second Ubuntu server virtual machine for our next task – setting up Immich, Onlyoffice, and Syncthing. These instructions are virtually identical to the instructions for installing a virtual machine for Mailcow.

<span id="what-makes-this-virtual-machine-installation-different-from-mailcows-vm-installation"></span>
=== What makes this virtual machine installation different from Mailcow’s VM installation? ===

We want more RAM &amp; CPU power for this instance because:

* Immich is going to transcode videos we upload to video proxies
* Immich is going to run machine learning tasks on your photos (LOCALLY)
* Immich is going to create thumbnails of our photos

<blockquote>Note: What is a video proxy? Video proxies &amp; photo thumbnails are smaller, more compressed versions of the original video or picture that allow you to load them quickly even when your internet connection is slow.
</blockquote>
<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:lu55028jxef6_tmp_a05cc5c9.png
</gallery>

<span id="step-1-setting-up-virtual-machine-manager-virsh-1"></span>
=== Step 1: Setting up Virtual Machine Manager (virsh) ===

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:lu55028jxef6_tmp_b51c10bd.png
</gallery>

# In '''Virtual Machine Manager''', click '''File &gt; New Virtual Machine''' from the menu.

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:lu55028jxef6_tmp_151745e3.png
File:lu55028jxef6_tmp_9a251511.png
File:lu55028jxef6_tmp_8bf9cc92.png
</gallery>

<span id="choose-installation-media-1"></span>
==== 1.1 Choose Installation Media ====

* Select '''“Local install media (ISO image or CDROM)”''' and click '''“Forward”'''.
* Click '''“Browse”''' to select your Ubuntu Server ISO.
* Choose the ISO file you prepared earlier (e.g., <code>/var/lib/libvirt/images/ubuntu-server.iso</code>) and click '''“Forward”'''.

<span id="choose-operating-system-version-1"></span>
==== 1.2 Choose Operating System Version: ====

* Virtual Machine Manager may automatically detect the OS. If not, search for <code>ubuntu</code> and choose what is closest to your version. When in total doubt, <code>linux generic 2022</code> works.
* Click'''“Forward”'''.

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:lu55028jxef6_tmp_f72805e.png
</gallery>

<span id="configure-memory-and-cpu-1"></span>
==== 1.3 Configure Memory and CPU: ====

* Allocate the resources for your VM:
** '''Set RAM''': I would use at LEAST 75% of your machine’s RAM.
** '''Set vCPUs''': I would set this to at least 75% of your CPU’s cores.
* Click “Forward”.

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:lu55028jxef6_tmp_52d81284.png
</gallery>

<span id="configure-storage-1"></span>
==== 1.4 Configure Storage: ====

* Select '''Create a disk image for the virtual machine'''.
* I would make this as large as you imagine your entire smartphone backup to be, plus extra for padding.
* What is the size of ALL of your photos, videos, and files on your phone? That’s the size to choose here.
* '''When I say videos, I do not mean things you want to watch at home/on your TV – we will have another setup for that. I mean your personal photo albums/videos recorded on your phone.'''
* Make sure the disk image format is QCOW2. This format supports resizing, and other cool features.
* Click '''“Forward”'''.

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:lu55028jxef6_tmp_f64d0065.png
</gallery>

<span id="set-up-networking-with-the-bridge-interface-1"></span>
==== 1.5 Set Up Networking with the Bridge Interface ====

* Choose '''“Specify shared device name”''' under '''“Network Selection”'''.
* In the Device Name field, type <code>br0</code> (or whatever name you have given your bridge interface).
* This will allow the VM to grab a static IP from the same network as your host machine, making sure it acts like an independent hardware device.
* Click '''“Forward”'''.

<div class="figure">

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:lu55028jxef6_tmp_60757ece.png
</gallery>

</div>
<span id="finish-customize-before-installing-1"></span>
==== 1.6 Finish &amp; Customize Before Installing ====

* Name your virtual machine (e.g., “androidstuff”), something suitable for what this machine will do.
* Before clicking '''“Finish”''', check the box that says '''“Customize configuration before install”'''.
* Click '''“Finish”'''.

<div class="figure">

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:lu55028jxef6_tmp_f5a5b1fa.png
</gallery>

</div>
<span id="step-2-install-ubuntu-server-as-a-virtual-machine-1"></span>
=== Step 2: Install Ubuntu Server as a Virtual Machine ===

'''I will be blazing through this since we did this already once - refer to Installing Ubuntu Server with RAID 1, LVM, and LUKS Encryption above.'''

'''Keep in mind the following:'''

We are NOT using LUKS encryption here. There is no need since the image is going to be stored on an encrypted partition.

We are NOT using RAID – this is a disk image that is being stored on a RAID array, so we are not doing that.

We are configuring networking the same as we did before, but we will be using a '''''different''''' IP address!

<span id="start-the-installation-process-in-the-virtual-machine-1"></span>
==== 2.1 Start the installation process in the virtual machine ====

Choose your language and select '''“Try or install Ubuntu Server”'''.

Follow the installation prompts.

<div class="figure">

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:lu55028jxef6_tmp_ec4e8896.png
</gallery>

</div>
<div class="figure">

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:lu55028jxef6_tmp_578acc67.png
</gallery>

</div>
<span id="configure-static-ip-address-1"></span>
==== 2.2 Configure Static IP Address ====

* When you reach the Network configuration screen, select the network interface that corresponds to your network interface.
* Choose the option '''“Configure network manually”'''.
* Enter the following details:
** IP Address: '''192.168.5.5'''
** Subnet: '''192.168.5.0/24'''
** Gateway: '''192.168.5.1'''
** Nameserver: '''192.168.5.1'''
* Make sure you enter all the details correctly to provide the virtual machine has the correct static IP configuration.

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:lu55028jxef6_tmp_96b658d9.png
File:lu55028jxef6_tmp_db73416d.png
File:lu55028jxef6_tmp_395bacbb.png
File:lu55028jxef6_tmp_11e6c2bf.png
File:lu55028jxef6_tmp_b4d42965.png
File:lu55028jxef6_tmp_c3cac3b1.png
File:lu55028jxef6_tmp_d358ecfb.png
File:lu55028jxef6_tmp_35442bae.png
File:lu55028jxef6_tmp_42a93091.png
File:lu55028jxef6_tmp_f21e3690.png
File:lu55028jxef6_tmp_220bed13.png
File:lu55028jxef6_tmp_36760d5c.png
File:lu55028jxef6_tmp_eb8a4fe0.png
File:lu55028jxef6_tmp_64120bbe.png
File:lu55028jxef6_tmp_8281f281.png
File:lu55028jxef6_tmp_b6f1ac1f.png
File:lu55028jxef6_tmp_89692e7.png
</gallery>

<span id="partition-the-virtual-drive-1"></span>
==== 2.3 Partition the virtual “drive” ====

* When you reach the Filesystem setup section, select “Use an entire disk” and then choose the disk you want to install Ubuntu Server on.
* Choose the option “Set up this disk as an LVM group”.
* '''Important:''' At this stage, edit the partition sizes as Ubuntu’s installer usually allocates 2 GB for boot which is ridiculous and even worse it only uses half the available space for your LVM &amp; root. The Ubuntu auto partitioner is horrible.
* Reduce the boot partition to 512 MB.
* Delete the old LVM &amp; root partition.
* Create a new LVM taking up the entire disk.
* Create a logical volume for the root filesystem, using all available space.
* '''Do not encrypt the volume''' (it’s unnecessary since the host drive is already encrypted, and it is not my intention for you to have these VMs running on other people’s servers).

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:lu55028jxef6_tmp_3d6c5298.png
File:lu55028jxef6_tmp_751040c0.png
</gallery>

<span id="finalize-installation-do-not-install-docker-1"></span>
==== 2.4 Finalize installation &amp; do not install docker ====

* Set up your username and password.
* '''Choose to install OpenSSH server.'''

<blockquote>'''WARNING:''' DO NOT CHOOSE TO INSTALL DOCKER USING THE PROMPT AFTER THIS!
</blockquote>
* After configuring the partition sizes, proceed with the installation process as usual, following the prompts to set up any additional software you want to install.
* Once the installation is complete, the system will automatically apply your network &amp; partitioning settings.
* When prompted, remove the installation media (ISO) disk image from the virtual machine settings.
* Restart the virtual machine.

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:lu55028jxef6_tmp_ce86cb27.png
File:lu55028jxef6_tmp_b291175e.png
File:lu55028jxef6_tmp_b1c36016.png
</gallery>

<span id="remove-the-cdrom-1"></span>
==== 2.5 Remove the CDROM ====

* Go to '''View —&gt; Details''' in '''Virtual Machine Manager'''
* Go to '''“SATA CDROM”''' on the left side.
* Confirm that the '''“source path”''' is the ubuntu iso we downloaded for installing Ubuntu server on this virtual machine
* Click '''“remove”''' in the lower right corner.
* UNCHECK '''“delete associated storage files”''' – we will use this image again later!
* Click delete.
* You may have to turn off the VM to do this.

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:lu55028jxef6_tmp_fb68028c.png
File:lu55028jxef6_tmp_28adba83.png
File:lu55028jxef6_tmp_85359533.png
File:lu55028jxef6_tmp_74771b5d.png
File:lu55028jxef6_tmp_83560fb2.png
File:lu55028jxef6_tmp_f5637068.png
File:lu55028jxef6_tmp_e789b00f.png
</gallery>

<span id="set-up-static-ip-mapping-in-pfsense-2"></span>
==== 2.6 Set Up Static IP Mapping in pfSense: ====

* Log into your '''pfSense''' router.
* Go to '''Status &gt; Diagnostics &gt; ARP Table'''.
* Find the MAC address associated with your server’s IP (in our case this is, '''192.168.5.5'''), copy it.
* Go to '''Services &gt; DHCP Server'''.
* Scroll to the bottom and click '''“Add Static Mapping”'''.
* Enter the MAC address and IP address of your server.
* Give it a descriptive name (such as “androidstuff static IP”).
* Set the hostname to <code>androidstuff</code>.
* Save and apply changes.

<blockquote>'''Note:''' This makes sure that this IP address is reserved for this computer to connect to, so that no other device can take it (unless they are spoofing MAC addresses, but if someone does, that’s a different story).
</blockquote>
<span id="set-up-this-virtual-machine-to-start-at-boot-1"></span>
==== 2.7 Set up this virtual machine to start at boot: ====

Type the following into the terminal at <code>happycloud</code>, which is our main server that we are creating all of these virtual machines on at <code>192.168.5.2</code>:

<pre>virsh autostart androidstuff</pre>
* Check that this is set up properly by typing <code>virsh dominfo androidstuff</code> and seeing if the autostart line is set to enable.
* If you don’t do this, you will realize once it is too late &amp; you’ve left your house after you have rebooted your server (for whatever reason) that none of your services are working. This will suck.
* This command makes it so that the virtual machine starts each time we boot the computer.

You’ve now successfully set up an '''Ubuntu Server''' virtual machine using Virtual Machine Manager, configured with a static IP address and LVM partitioning. We have a virtual machine that we just created that we can use to set up our second server for android backups, image search using machine learning &amp; face detection with local models that don’t connect to the internet. '''EXCITED'''??? I AM! :D :D :D

<span id="step-2-setting-up-syncthing-for-android-backups"></span>

== Step 2: Setting up Syncthing for android backups ==

<span id="step-1-install-syncthing"></span>
=== Step 1: Install syncthing ===

<span id="add-the-syncthing-repository"></span>
==== 1.1 Add the Syncthing Repository ====

First, we need to add the Syncthing repository and its PGP key for package verification.

<ol style="list-style-type: decimal;">
<li><p>Create a directory for the keyring:</p>
<pre>sudo mkdir -p /etc/apt/keyrings</pre></li>
<li><p>Download the Syncthing release PGP key:</p>
<pre>sudo curl -L -o /etc/apt/keyrings/syncthing-archive-keyring.gpg https://syncthing.net/release-key.gpg</pre></li>
<li><p>Add the Syncthing stable repository to your APT sources:</p>
<pre>echo &quot;deb [signed-by=/etc/apt/keyrings/syncthing-archive-keyring.gpg] https://apt.syncthing.net/ syncthing stable&quot; | sudo tee /etc/apt/sources.list.d/syncthing.list</pre></li></ol>

<span id="make-sure-syncthing-repository-takes-priority"></span>
==== 1.2 Make Sure Syncthing Repository Takes Priority ====

To make sure the system packages don’t take preference over the ones in the Syncthing repository:

<ul>
<li><p>Create a preferences file for APT:</p>
<pre>sudo nano /etc/apt/preferences.d/syncthing</pre></li>
<li><p>Add the following content to the file:</p>
<pre>Package: *
Pin: origin apt.syncthing.net
Pin-Priority: 990</pre></li>
<li><p>Save &amp; exit the editor (in nano, press <code>Ctrl+X</code>, then <code>Y</code>, then <code>Enter</code>).</p></li></ul>

<span id="install-syncthing"></span>
==== 1.3 Install Syncthing ====

Now that we’ve added the repository and made sure its priority, let’s install Syncthing:

<ul>
<li><p>Update the package lists and make sure your system is up to date:</p>
<pre>sudo apt-get update
sudo apt-get upgrade -y</pre></li>
<li><p>Install Syncthing:</p>
<pre>sudo apt-get install syncthing -y</pre></li></ul>

<span id="step-2-setting-up-syncthing-as-a-system-service"></span>
=== Step 2: Setting Up Syncthing as a System Service ===

To have Syncthing start automatically on system boot, even without user login, we’ll set it up as a <code>systemd</code> service that runs as our user, even if we haven’t logged in yet.

<span id="create-a-systemd-service-file"></span>
==== 2.1 Create a Systemd Service File ====

<ol style="list-style-type: decimal;">
<li><p>Create a new service file:</p>
<pre>sudo nano /etc/systemd/system/syncthing@$USER.service</pre></li>
<li><p>Add the following content to the file:</p>
<pre>[Unit]
Description=Syncthing
Documentation=man:syncthing
After=network.target

[Service]
User=%i
ExecStart=/usr/bin/syncthing -no-browser -gui-address=0.0.0.0:8384
Restart=on-failure
RestartSec=5
SuccessExitStatus=3 4
RestartForceExitStatus=3 4
# Harder
ProtectSystem=full
PrivateTmp=true
SystemCallArchitectures=native
MemoryDenyWriteExecute=true
NoNewPrivileges=true

[Install]
WantedBy=multi-user.target</pre></li>
<li><p>Save and exit the editor, hit <code>Ctrl+X</code> then <code>Y</code> to save.</p></li></ol>

<span id="configure-the-service"></span>
==== 2.2 Configure the Service ====

<ol style="list-style-type: decimal;">
<li><p>Enable the service:</p>
<pre>sudo systemctl enable syncthing@$USER.service</pre></li>
<li><p>Start the service:</p>
<pre>sudo systemctl start syncthing@$USER.service</pre></li></ol>

<span id="step-3-securing-syncthings-web-interface"></span>
=== Step 3: Securing Syncthing’s Web Interface ===

By default, Syncthing’s web interface is accessible from any device that can reach your server. This makes it very important to secure the interface with a strong password.

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:lu55028jxef6_tmp_a1d1627a.png
File:lu55028jxef6_tmp_7548e599.png
File:lu55028jxef6_tmp_79b6bf05.png
File:lu55028jxef6_tmp_e4e01235.png
File:lu55028jxef6_tmp_3e71e99.png
</gallery>

<span id="access-the-web-interface-1"></span>
==== 3.1 Access the Web Interface ====

# Open a web browser and navigate to <code>http://192.168.5.5:8384</code> or <code>http://androidstuff.home.arpa</code>.
# You should see the Syncthing web interface.

<span id="add-a-gui-password"></span>
==== 3.2 Add a GUI Password ====

# In the web interface, click on the “Actions” button (gear icon) in the top right corner.
# Select '''“Settings”''' from the dropdown menu.
# In the Settings page, scroll down to the '''“GUI”''' section.
# Find the '''“GUI Authentication User”''' field and enter a username.
# In the '''“GUI Authentication Password”''' field, enter a strong password.
# Check '''“Use HTTPS for GUI”''' so we can visit the server using https://androidstuff.home.arpa:8384 instead. It’s a good habit. :)

<blockquote>'''Note:''' Choose a complex password so some random person who attaches to your home wifi if you forget to set up a guest network that has no LAN access can’t mess with your Syncthing configuration.
</blockquote>
<ol start="7" style="list-style-type: decimal;">
<li>Click '''“Save”''' at the bottom of the page.</li>
<li>Syncthing will prompt you to confirm the changes. Click '''“Yes”''' to apply the new settings.</li>
<li>You’ll be logged out and prompted to log in with your new credentials.</li>
<li>Attempt to access the interface again. You should be prompted for the username and password you set. If not, you messed something up. Do not pass go, do not collect $200, until this asks you for a password to log in.</li></ol>

<span id="step-4-configuring-syncthing-discovery-settings"></span>

=== Step 4: Configuring Syncthing Discovery Settings ===

<span id="understanding-discovery-methods-why-we-dont-use-them."></span>
==== 4.1 Understanding Discovery Methods &amp; why we DON’T USE THEM. ====

Discovery methods are how the syncthing app on your phone will “find” the server you set up as your backup server.

<blockquote>'''NOTE:''' Our server has a static IP: <code>192.168.5.5</code>. We went through the trouble to make sure it always lives at <code>192.168.5.5</code> via static mappings in '''pfSense''' and configuring a static IP in the server’s networking settings. Our server will '''always''' be present at <code>192.168.5.5</code> or <code>androidstuff.home.arpa</code> while we are connected via VPN. All Syncthing “discovery” is doing is trying to find our machine, but why use a find feature when we already know where it is? This adds another point of failure for no good reason! Think of it like making your iPhone invisible &amp; then enabling '''“find my iPhone.”'''
</blockquote>
This setup we are installing syncthing onto has the following:

# A static IP configured, so that it is '''always''' <code>192.168.5.5</code>
# A static IP mapping configured in our router, so that no other device on our network can ever steal <code>192.168.5.5</code> from the computer running syncthing.
# A static hostname of <code>androidstuff</code> that does not change.
# Dynamic DNS for our main internet connection, so when we are outside our network our '''pfSense''' router &amp; <code>FreeDNS</code> will make sure that <code>louishomeserver.chickenkiller.com</code> always points to our home network IP address.

'''I will showcase local discovery failing on video.''' It ''“works”'' when I initially connect to my server via QR code &amp; visiting it in the browser, but fails when I try to connect again. This is because my VPN is on network <code>192.168.6.0/24</code> and my Syncthing is on <code>192.168.5.0/24</code>. I was hoping local discovery would be “smart” enough to remember the last IP address my server was on since it had not changed, but it did not.

'''NEVER RELY ON SOMETHING ELSE TO BE “SMART” IN SOLVING A PROBLEM THAT DOES NOT HAVE TO EXIST IN THE FIRST PLACE!'''

<span id="local-discovery-do-not-trust"></span>

==== 4.2 Local Discovery – DO NOT TRUST! ====

Local discovery allows Syncthing to find other devices on your local network automatically. Key word, ''local'' – meaning your subnet of <code>192.168.5.0/24</code>. What if you connect via your VPN, which is on <code>192.168.6.0/24</code>?

When we first add the QR code of our Syncthing instance to our Android phone Syncthing app, Syncthing will connect to our desktop server running Syncthing. HOWEVER: our Android application will NOT find the Syncthing server the NEXT time we connect. THIS IS BAD!!

This is even worse than it not working at all, as it will give the false impression that it works. This is how people who have set up “backup solutions” end up as customers of Rossmann Repair Group paying $2000 to recover a hard drive that fell off a balcony.

<span id="connecting-reliably-to-syncthing-without-discovery-hassles"></span>
===== Connecting Reliably to Syncthing without Discovery Hassles =====

This situation is actually '''worse''' than if Syncthing had no Local Discovery feature at all. If it didn’t work from the start, you’d know you couldn’t rely on it and would just hardcode the IP of your Syncthing server right into your Android app, using the server’s local IP to connect directly.

What’s dangerous is that Syncthing’s Android app connects the first time by scanning the QR code on the server, making it seem like it’s actually discovering your computer. But it’s not. Next time you try to connect—especially if you’re on a different subnet via VPN—it’ll fail to find the server.

'''Syncthing doesn’t even remember the last IP address it used, so it ends up trying to rediscover it, failing again.'''

I get it. If it can’t find the server on a different subnet when you’re using a VPN, fine, but it’s dangerous that Syncthing doesn’t try the last known IP to see if it still works.

'''TL;DR – to avoid becoming a data recovery customer, don’t trust local or global discovery. Just use the IP address of the server, which in our case is <code>192.168.5.5</code>, and check that it works three separate times under three separate conditions before ever assuming that it is working, as you should with ANY backup solution!'''

<span id="global-discovery"></span>
==== 4.3 Global Discovery ====

Global discovery helps Syncthing find your devices over the internet. It works by periodically announcing your device’s presence to global discovery servers.

* '''Privacy Implications:''' Higher risk, as it involves sharing your device’s information with external servers. This could potentially expose:
** Your IP address
** The fact that you’re using Syncthing
** When your device is online

The bigger issue with this is not privacy, ''it’s that it is unnecessary'' and adds another point of failure over entering the hostname manually.

<span id="configuring-discovery-settings"></span>
==== 4.2 Configuring Discovery Settings ====

# '''Access Syncthing Settings'''
## Open the Syncthing web interface (typically <code>https://192.168.5.5</code> or <code>https://androidstuff.home.arpa:8384</code>).
## Click on the “Actions” button (gear icon) in the top right corner.
## Select “Settings” from the dropdown menu.
# '''Adjust Discovery Settings'''
## In the Settings page, scroll to the “Connections” section.
## Find the following options:
##* '''Enable Local Discovery:''' Keep this checked.
##* '''Enable Global Discovery:''' Uncheck this box.
## Click “Save” at the bottom of the page.
## Syncthing will prompt you to confirm the changes. Click “Yes” to apply the new settings.

<span id="step-5-connecting-server-syncthing-to-android-syncthing"></span>
=== Step 5: Connecting server syncthing to android syncthing ===

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:lu55028jxef6_tmp_90112fd.png
File:lu55028jxef6_tmp_5f13fe67.png
File:lu55028jxef6_tmp_98846264.png
File:lu55028jxef6_tmp_7cd671c.png
File:lu55028jxef6_tmp_9884f00d.png
File:lu55028jxef6_tmp_1527f750.png
File:lu55028jxef6_tmp_27000b93.png
File:lu55028jxef6_tmp_2a5ef23.png
File:lu55028jxef6_tmp_66c6b48d.png
</gallery>

<span id="connect-to-your-vpn."></span>
==== 5.0 – Connect to your VPN. ====

Your android phone must be connected to your VPN for you to connect to your server if your phone is not on the same wifi network as the virtual machine running the syncthing server.

<span id="install-syncthing-from-the-f-droid-store."></span>
==== 5.1 Install syncthing from the f-droid store. ====

* Go to the ''[https://f-droid.org/en/packages/com.github.catfriend1.syncthingandroid/ F-Droid Store to install syncthing-fork]''
* Upon starting syncthing, provide it permissions for notifications.
* Permissions for location are not necessary.

<span id="avoid-becoming-a-data-recovery-customer"></span>

==== 5.2 Avoid becoming a data recovery customer ====

'''Delete the Camera Folder''': Not from the device, just from the sync list, within syncthing. Tap on the camera folder &amp; hit the trash bin in the upper right.

There’s a good reason for that. You might think, “Why? I WANT to sync and back up my photos and videos!!” Here’s the thing: sometimes, camera apps switch folders without you knowing. I’ve seen cases where photos were saved in a different folder INSIDE the DCIM folder, and the gallery app only showed one specific folder.

I’m not a predatory technician that ''[https://www.youtube.com/watch?v=OVZTBhVV5tI&pp=ygUVZHJpdmVzYXZlcnMgIHJvc3NtYW5u bills people $3000 for a bad iPhone screen or charge port]''. But they are out there, and someone was close to paying $500 to a different scam artist data recovery company because their gallery app wasn’t checking a 2nd folder inside of the DCIM folder where another program was saving photos to.

We are not going to back up the camera folder ''within'' the DCIM folder. We are going to back up the '''entire DCIM folder.'''

For those who don’t know, on 99% of Android devices, '''DCIM''' is a folder in the root directory of the ''“visible”'' filesystem within which the subfolders storing your recorded videos &amp; pictures reside.

Next, I am going to do something different. I wanted to show you what happens when you use local discovery/dynamic rather than inserting your actual server IP address into the server field. This meant including screenshots from a LATER step, after I had already added folders that we are going to sync, to show you how syncthing fails with local discovery. '''It’s important to me that you understand how this fails with images for yourself, so you don’t create a setup that makes you a data recovery customer.'''

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:lu55028jxef6_tmp_621c170c.png
File:lu55028jxef6_tmp_fb6bf453.png
File:lu55028jxef6_tmp_364a837e.png
File:lu55028jxef6_tmp_d9c45480.png
File:lu55028jxef6_tmp_1948d4b.png
File:lu55028jxef6_tmp_2bfff860.png
File:lu55028jxef6_tmp_2a5c001f.png
</gallery> hEREEEEEEEEEEEEEE<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:lu55028jxef6_tmp_cc3f5925.png
File:lu55028jxef6_tmp_62d8fd43.png
File:lu55028jxef6_tmp_f74f4252.png
File:lu55028jxef6_tmp_cb876287.png
File:lu55028jxef6_tmp_123b314.png
File:lu55028jxef6_tmp_4e264fbe.png
</gallery>

'''Here is what will happen if you set this up with dynamic, disconnect, and then reconnect. Note how it shows up as “idle” for syncing and “disconnected” on the android phone; it is transferring NOTHING, even though the desktop syncthing server GUI shows that we are out of sync.'''

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:lu55028jxef6_tmp_b52ee6c4.png
File:lu55028jxef6_tmp_30c1edaf.png
</gallery>

<span id="add-a-device-to-syncthing-android-app"></span>
==== 5.3 Add a device to syncthing android app ====

# On the top, you’ll see '''Folders''' and '''Devices'''.
# Tap '''Devices'''.
# Tap the plus in the upper right corner to add a device.
# Tap the QR code next to '''Device ID''' in the upper right.
# Go back to the '''Ubuntu Server Syncthing Web Interface'''.
#* Open a web browser and navigate to <code>http://192.168.5.5:8384</code> or [http://androidstuff.home.arpa:8384/ http://androidstuff.home.arpa:8384].
# Obtain Device ID and QR Code
#* In the web interface, click on the blue gobbledygook of numbers &amp; letters next to '''“Identification”''' under '''“This Device”''' (gear icon) in the top right.
#* Select '''“Show ID”'''.
#* You’ll see a QR code and the device ID. ''SCAN YOURS. DO NOT SCAN MINE. I SHOWED A PICTURE OF MINE SO YOU CAN SEE WHAT IT LOOKS LIKE.''
# Configure Device Settings on Android
#* '''Device Name:''' Enter a recognizable name (e.g., “Ubuntu Server”).
#* '''Addresses:''' ''DO NOT CHOOSE DYNAMIC. USING DYNAMIC WILL CAUSE IT TO NOT SYNC WHEN YOU DISCONNECT &amp; RECONNECT FROM YOUR NETWORK. IT WILL WORK THE FIRST TIME, AND THEN NEVER SYNC AGAIN, AND YOU WILL BE PAYING DATA RECOVERY DOUCHEBAGS TO RECOVER YOUR PHONE.''

<blockquote>'''How dynamic failed:''' I used “dynamic” as an example of why it doesn’t make sense to use autodiscovery when you KNOW where your server is. I chose dynamic, and it connected &amp; worked. When I disconnected from my network &amp; reconnected, the ''Devices'' tab in the Syncthing Android app showed me to be ''disconnected'' and the ''Folders'' tab showed the folders to be ''idle'' even though the web GUI for Syncthing said that my folder was ''Out of sync'' and ''Remote Devices'' showed my phone as ''Disconnected''.
</blockquote>
<ul>
<li><p>FILL IN '''“Address”''' when adding a device as follows, if you used the setup I was using within this guide to Syncthing.</p>
<pre>tcp://192.168.5.5:22000</pre></li>
<li><p>OR</p>
<pre>tcp://androidstuff.home.arpa:22000</pre></li>
<li><p>The format is <code>tcp://</code>, then your IP address, then <code>:22000</code> for the port.</p></li>
<li><p>No need to check “Introduce new devices”.</p></li>
<li><p>'''Did you include the <code>tcp://</code> at the beginning, and the <code>:22000</code> at the end for the port? You’d better have!'''</p></li>
<li><p>Save and continue.</p></li></ul>

<ol start="8" style="list-style-type: decimal;">
<li><p>'''Approve the Connection on Ubuntu Server'''</p>
<ul>
<li><p>Return to the Ubuntu Server web interface.</p></li>
<li><p>You should see a prompt to add a new device.</p></li>
<li><p>Verify the Device ID matches your Android device.</p></li>
<li><p>Click '''“Add Device”'''.</p></li>
<li><p>Set a name for the Android device (e.g., “Android Phone”).</p></li>
<li><p>Click '''“Save”'''.</p></li></ul>
</li>
<li><p>'''Check the Connection'''</p>
<ul>
<li>On both devices, check that the other device appears as connected. The connection might take a few moments to establish.</li></ul>
</li></ol>

<blockquote>'''Note:''' Make sure that port <code>22000</code> (or your configured Syncthing port) is open in your Ubuntu Server’s firewall for incoming connections from your local network. B'''y default <code>ufw</code> is not running and blocking things when you first boot Ubuntu Server''' but that may change at a later date, same way they snuck in the suggestion of pre-installing a snap version of Docker.
</blockquote>
Now you’ve added your Ubuntu Server Syncthing instance to your phone; no open ports, will sync whenever you are on wifi with your VPN on, and continuously back up your phone. Beautiful. :)

<blockquote>'''REMEMBER – DO NOT SET “ADDRESSES” TO “DYNAMIC” – TAP “DYNAMIC” AND REPLACE IT WITH''' <code>tcp://youripaddress:22000</code> '''REPLACING “youripaddress” WITH THE IP ADDRESS OF THE VIRTUAL MACHINE THAT IS RUNNING SYNCTHING.'''
</blockquote>
<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:lu55028jxef6_tmp_65626c83.png
File:lu55028jxef6_tmp_135ac5ce.png
File:lu55028jxef6_tmp_ffc850bf.png
File:lu55028jxef6_tmp_23122602.png
File:lu55028jxef6_tmp_4468da5a.png
File:lu55028jxef6_tmp_a83ae883.png
</gallery>

<span id="step-6-configuring-syncthing-for-organized-android-backups"></span>
=== Step 6: Configuring Syncthing for Organized Android Backups ===

<span id="configure-android-syncthing-app"></span>
==== 6.1 Configure Android Syncthing App ====

# Open Syncthing on your Android device.
# For each folder you want to sync:
#* Tap the plus icon in the upper right in the folders part of the app.
#* Tap folder label and label it.
#* Tap the directory and choose your directory you want to sync (it’ll let you choose everything besides the download folder on android).
# '''MAKE SURE TO TOGGLE THE SERVER SWITCH UNDER WHERE YOU TAPPED TO CHOOSE THE DIRECTORY YOU WANTED TO SYNC SO THAT IT ACTUALLY BACKS UP.'''
#* Choose ''send &amp; receive'' if you want two-way folder sync.
#* Choose ''send'' if you want it to only send files to your server.
#* Choose ''receive'' if you only want it to receive files from your server

A good rule of thumb: For smaller folders and stuff you transfer to your phone to read on a trip, audiobooks, etc., I choose ''SEND &amp; RECEIVE'' so I can transfer both ways. For stuff like videos I record and photos I take (the DCIM folder), I choose ''SEND ONLY''. I have a 256 GB phone, and over 1.3 terabytes of videos I have recorded… I can’t sync all of that to my phone or it will fill up. But I have less than 1 GB of audiobooks, books, and max 20 GB of movies I am watching at any given time on my phone.

<ol start="7" style="list-style-type: decimal;">
<li>'''Tap checkbox in upper right corner when done.'''</li></ol>

<span id="syncing-on-wifi-only-yes-or-no"></span>
==== 6.2 Syncing on wifi only – yes or no? ====

Your Android device can connect to Syncthing, and you can configure Syncthing while you’re on the go. But by default, your Android device must be on wifi in order for file transfer and backup to occur. Even if you are connected to your VPN, your Android device is not going to transfer files if you are not on wifi.

The way you change this is by editing the folder settings in the Syncthing Android app, and disabling the “sync on wifi only” option. I would suggest doing this for folders with SMALL files like documents, audiobooks, and not for folders with LARGE files like the DCIM folder with your recorded videos and camera pictures.

Unlimited plans have data caps; try using 200 GB in 10 days on any ''“unlimited”'' wireless data plan in the United States and watch your ''“unlimited 5G”'' turn into a 56k modem. The only reason they can market using this wankery is because consumer protection law in the United States is a joke.

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:lu55028jxef6_tmp_32088dd0.png
File:lu55028jxef6_tmp_40872f6d.png
File:lu55028jxef6_tmp_5f537918.png
File:lu55028jxef6_tmp_5b05dae0.png
File:lu55028jxef6_tmp_a7b2093d.png
File:lu55028jxef6_tmp_16c65783.png
File:lu55028jxef6_tmp_19a75a7a.png
File:lu55028jxef6_tmp_ceaf4d3d.png
File:lu55028jxef6_tmp_9bd9aead.png
</gallery>

<span id="accept-folders-on-ubuntu-server"></span>
==== 6.3 Accept Folders on Ubuntu Server ====

# On the Syncthing web interface of your Ubuntu server, you’ll see notifications for new folders.
# For each folder: Click '''“Add”'''.
# CHANGE THE BASE DIRECTORY FROM <code>~/(foldernamehere)</code> '''to''' <code>~/androidbackup/(foldernamehere)</code> so you don’t clog up your base directory. This makes it easy to see in one click what everything we’re backing up from the android phone is.
# Click '''Save'''.

<span id="creating-new-folders-on-ubuntu-server"></span>
==== 6.4 Creating New Folders on Ubuntu Server ====

* It does it for you. What a beautiful program, right? :)

<span id="step-7-verify-and-test-inspect-what-you-expect"></span>
=== Step 7: Verify and Test – INSPECT WHAT YOU EXPECT! ===

'''Don’t become a data recovery customer. Syncthing is used for backing up your phone - arguably the most important part of this entire process.'''

# 99% of the people who show up for data recovery at a data recovery business thought their data was backing up.
# '''It was not.'''
# Use common sense, look through the folders on your server, look at the web interface, make sure things open.

'''You now have working Android backups!'''

* All folders from your Android device will be organized within the <code>~/androidbackup</code> directory.
* Each Android folder will have its own subdirectory for better organization.

<span id="step-3-installing-onlyoffice-workspace-and-wsgidav-to-replace-google-docs"></span>
== Step 3: Installing ONLYOFFICE Workspace and WsgiDAV to replace Google Docs ==

So we have Syncthing, but how do we edit documents we have on our backup server? SSH in? vi? nano?

No.

We are going to use the same virtual machine for this that we used for Syncthing and install something called ONLYOFFICE.

<span id="nextcloud"></span>
=== Nextcloud? ===

The first thing many people are going to suggest is Nextcloud. Nextcloud is that all-in-one cloud suite that will change your contacts from read-write to read-only so that your contacts get deleted when you update ''(without telling you, of course)'', that ''[https://help.nextcloud.com/t/calendar-timezone-bug/178056 can’t tell time]''. Might it surprise you if I told you that it is miserably slow, and that it gave errors unless you clicked a separate submenu to open a document?

<span id="moving-to-onlyoffice"></span>
=== Moving to OnlyOffice ===

OnlyOffice is fast, and it is used by people who actually pay them. This means that their software has to work, and it does!

<span id="step-0-install-docker-properly."></span>
=== Step 0: Install docker properly. ===

<span id="never-use-ubuntus-snap-version-of-docker"></span>
==== Never use Ubuntu’s snap version of docker ====

Ubuntu installs docker by default using the cancerous snap. We do not want to use snap. Ubuntu installer will ask if you want to install Docker, and you should always say No. 

<span id="doesnt-onlyoffices-install-script-install-docker-for-me"></span>
==== Doesn’t onlyoffice’s install script install docker for me? ====

Onlyoffice’s installation script '''DOES''' install docker for you. I am still going to have you do it manually.

* If you choose to not install onlyoffice, and wish to install Immich, I want you to know how to install docker on this virtual machine ''yourself.''
* I don’t want to rely on onlyoffice’s script. It won’t install docker for us if it detects Docker already, so we’re not going to do a double install. What if onlyoffice’s installation script stops installing docker the same way in a new version, or stops installing docker at all within its script?

It’s little work to install Docker the right way for our purposes manually, and it’s good to have it documented so that you can use docker for immich even if you elect not to install Onlyoffice.

<span id="update-and-upgrade-your-system-1"></span>
==== 0.1 Update and upgrade your system ====

<pre>sudo apt update &amp;&amp; sudo apt upgrade -y
sudo apt install curl git wget -y</pre>
<span id="check-for-other-docker-installations-1"></span>
==== 0.2 Check for other Docker installations: ====

Run <code>docker --version</code> and see what is installed. Nothing should be installed yet since this is a fresh system. If something is installed, remove it.

<pre># Just in case you accidentally installed snap version of docker:

sudo snap remove docker

For other versions of docker: 

sudo apt remove docker docker-engine docker.io containerd runc</pre>
<span id="install-docker-using-official-docker-script-1"></span>
==== 0.3 Install Docker using official Docker script: ====

<pre>curl -fsSL https://get.docker.com -o get-docker.sh
sudo sh get-docker.sh</pre>
<blockquote>'''Note:''' It’s very important to use the official Docker installation and not the Snap version. The Snap version can cause issues due to its sandboxed nature, making it a mess for mailcow’s requirements. Docker snap makes me sad, and it’ll make you sad too if you try to make things work with it.
</blockquote>
<span id="install-docker-compose-1"></span>
==== 0.4 Install Docker Compose: ====

Ubuntu’s <code>docker-compose-plugin</code> is safe to use, it is not snap cancer.

<pre>sudo apt install docker-compose-plugin -y
sudo systemctl enable --now docker</pre>
<span id="verify-the-install-1"></span>
==== 0.5 Verify the install ====

Run <code>docker compose version</code> and make sure the version is 2.0 or higher. Run <code>docker --version</code> and make sure version is 24.0.0 or higher

<span id="set-proper-permissions-1"></span>
==== 0.6 Set proper permissions: ====

Docker needs to be run as root for some operations, but you can add your user to the docker group to avoid using <code>sudo</code> all the time. To be clear, mailcow’s own [https://docs.mailcow.email/getstarted/install/#check-selinux-specifics documentation] and [https://community.mailcow.email/d/59-mailcow-containers-running-as-root community] suggest starting with root or <code>sudo</code>, and you should trust them more than me. To quote mailcow developers, ''“Controlling the Docker daemon as non-root user does not give you additional security. The unprivileged user will spawn the containers as root likewise. The behaviour of the stack is identical.”'' Run this command to add your user:

<pre>sudo usermod -aG docker $USER</pre>
Log out and log back in, or run:
<code>newgrp docker</code>

<span id="step-1-install-onlyoffice-workspace-community-edition"></span>
=== Step 1: Install ONLYOFFICE Workspace Community Edition ===

It is very important that you follow the right steps. OnlyOffice’s website is a minefield of documentation that will lead to broken installations like this, even if you follow their instructions:

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:lu55028jxemg_tmp_3797c4d2.png
</gallery>

OR

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:lu55028jxemg_tmp_4b463750.png
File:lu55028jxemg_tmp_6712c2a2.png
File:lu55028jxemg_tmp_36c5237e.png
File:lu55028jxemg_tmp_a479215.png
</gallery>

You’re going to avoid the open-source hellscape above, by installing like this:

<ol style="list-style-type: decimal;">
<li><p>SSH into the <code>androidstuff</code> virtual machine we created at <code>192.168.5.5</code></p>
<pre>ssh username@192.168.5.5</pre></li>
<li><p>'''Download the ONLYOFFICE Workspace installation script:'''</p>
<pre>wget https://download.onlyoffice.com/install/workspace-install.sh</pre></li>
<li><p>'''Make the script executable:'''</p>
<pre>chmod +x workspace-install.sh</pre>
<p>This changes the file permissions to allow execution.</p></li>
<li><p>'''Run the installation script:'''</p>
<pre>sudo bash workspace-install.sh -it WORKSPACE -md fakedomainname.com</pre>
<p>Replace “fakedomainname” with your actual domain name from the mailcow section. You can also leave out <code>-md</code> and not install it.</p></li></ol>

<blockquote>'''CAUTION:''' Instructions within documentation on OnlyOffice website will lead to a broken installation. Use the command line above so it actually works.
</blockquote>
<ol start="5" style="list-style-type: decimal;">
<li><p>Once this is done, log in by going to [http://192.168.5.5/ http://192.168.5.5]</p></li>
<li><p>It will prompt you to make a username and a password. Go for it.</p></li>
<li><p>Once logged in, make an HTTPS SSL certificate so we can log in via HTTPS:</p></li></ol>

Go to '''Control Panel''', the big icon on the main home screen.

* Go to '''HTTPS''' on the top of the left menu.
* Click '''Generate and apply'''.
* Be happy.

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:lu55028jxemg_tmp_24ca3e8b.png
File:lu55028jxemg_tmp_d8250288.png
File:lu55028jxemg_tmp_9dc6bcc4.png
File:lu55028jxemg_tmp_b9441e34.png
File:lu55028jxemg_tmp_dd4bee59.png
File:lu55028jxemg_tmp_5206382b.png
File:lu55028jxemg_tmp_4a1d5bab.png
File:lu55028jxemg_tmp_7d1ddacb.png
File:lu55028jxemg_tmp_4468b1aa.png
File:lu55028jxemg_tmp_d27c50f2.png
File:lu55028jxemg_tmp_b3292451.png
</gallery>

<span id="step-2-local-file-access"></span>
=== Step 2: Local file access ===

Once you’re in, you’ll set up everything. Enter a password, agree to the terms of the license, and you’re good to go. I suggest entering administration settings and setting up HTTPS - it will make a self-signed certificate for you!

<span id="diving-into-open-sourcey-software"></span>
==== 2.1 Diving into “open sourcey” software ====

You can open a sample document. But what if I want this workspace server to be able to access ''files stored on the server??'' I want to open a document that’s on this computer; here’s where the fun begins. :)

<span id="the-rabbit-hole-to-hell-for-local-file-access"></span>
==== 2.2 The Rabbit Hole to hell for Local File Access ====

So, where do I go? There’s “Shared with me,” “Favorites,” “Reasons,” “Private room,” “Common in projects,” and “Invite users to Portal.” Maybe the settings? Let’s try that. Administrator profile settings, control panel… and oh, look, “Storage” - maybe I can add a local directory!!! … no, it’s all a mirage

<blockquote>'''Open Sourcism:''' You can’t just open a document from your server. It’s not a feature. You need to pass the direct URL to the document using a WebDAV server. Can you believe this? Welcome to the world of open source software!
</blockquote>
<span id="mounting-volumes-in-docker-failed-me"></span>
==== 2.3 Mounting volumes in Docker failed me ====

I went down the rabbit hole to figure this out when I tried weening myself off nextcloud a few years ago.

When you choose to install with Docker, there’s a script that gets downloaded. I explored the directory where this is installed—<code>onlyoffice</code>—and found the <code>document server</code>, <code>control panel</code>, <code>community server</code>, <code>MySQL setup</code>, and <code>mail server</code>.

In the <code>document server</code>, there’s a <code>data</code> directory. So, I thought, “Surely, I can mount it as a volume using Docker.” I searched for <code>:rw</code> to find where they’re specifying all the Docker volumes. It looks like a typical Docker Compose YAML file. I tried adding an argument for my directory, like <code>home/louis/Documents</code>, and mounted it in almost every possible location.

<blockquote>'''Important Note:''' The problem isn’t that the volume isn’t mounted. The issue is that this feature was never implemented in the software. They never thought a document server would need to access files on it. This is, again, the most open sourcey thing I’ve seen in a long time.
</blockquote>
<span id="fighting-open-source-winning"></span>
==== 2.4 Fighting open source &amp; winning ====

There’s a way to get files into this, but it won’t be immediately obvious. Going back to settings, there’s a menu called '''“Connected clouds”'''; we will use this to connect a WebDAV server to serve ourselves files.

We have to set up a webdav server, on our server, to serve files to the same virtual machine.

The whole idea of cloud server software is that you should be able to edit your documents in the cloud. No matter what computer you’re on, your files should be right there. But… my cloud server software can’t even read the files from my cloud server computer. Even if I mount those directories within the Docker volume, it still won’t work. The software wasn’t designed to see items in its own document data directory. But wait, it gets better.

<span id="the-solution"></span>
==== 2.5 The “Solution” ====

There’s a workaround for this. You can connect a new cloud. That you create, within your cloud. schrodinger’s cloud.

# Go through the settings and head to the control panel.
# You’ll see something called storage. You might think, “Oh, that’s where I can change things, right?” Wrong. There’s nothing there for connecting to local storage.
# Go back and find the connect button. It’s on the home screen under documents.
# Click “Connect” and we’re going to connect another cloud to our cloud.

We’re going to create a WebDAV server on our computer to feed files over to OnlyOffice. It’ll look like your directories are available, like it’s reading them off your computer, but we’re actually using WebDAV.

<span id="setting-up-webdav"></span>
==== 2.6 Setting Up WebDAV ====

We’re setting up a separate server to feed files to our server, on our server. There’s this small Python program called <code>wsgidav</code>. It’s a lightweight WebDAV server, not like setting up Apache or Nginx.

<span id="the-directory-problem"></span>
==== 2.7 The Directory Problem ====

Let’s say I want two directories: a <code>documents</code> directory and an <code>Android backup</code> directory. I can’t map both to WebDAV like you can in a Docker container. You can only log into one at a time.

Imagine having five different directories in one Docker volume but only being able to use ''one at a time.'' You’d have to log in differently each time.

You might think, ''“Louis, just create a new directory and symlink all the directories you want into it. What’s the problem?”'' Well, here’s where the open source rabbit hole goes deeper… the documentation for the software has an option called <code>follow symlinks</code>. You can set it to true, but it doesn’t work. Not unless you install a different version of the software because the version you get on PIP doesn’t work.

<blockquote>'''Warning:''' This will gaslight you to tears. You’ll pull your hair out wondering if you set up your symlinks right. It’s like a mirage—everything looks like it should work, but it doesn’t. I’m here to remind you that you are not insane.
</blockquote>
As Ralph Kramden would say, it doesn’t mean to be mean; it was just born that way.

I promise, this is all worth it to never have to use Nextcloud again. This is still better than Nextcloud, which tells you how bad Nextcloud is.

<span id="step-3-setting-up-a-webdav-server-on-linux"></span>
=== Step 3: Setting Up a WebDAV Server on GNU/Linux ===

<span id="install-and-configure-wsgidav"></span>
==== 3.1 Install and Configure WsgiDAV ====

WsgiDAV is a WebDAV server implementation written in Python.

<ol style="list-style-type: decimal;">
<li><p>'''Install WsgiDAV and its dependencies:'''</p>
<pre>sudo apt install python3-pip python3-dev libssl-dev libpam0g-dev -y
sudo pip3 install cheroot six python-pam
sudo pip install git+https://github.com/mar10/wsgidav.git</pre>
<blockquote><p>'''CAUTION:''' Do not install pip version of WsgiDAV as it will not work with the follow symlink option! These commands will install Python development files, SSL development files, WsgiDAV from github, and Cheroot (a WSGI server).</p></blockquote></li>
<li><p>'''Create WsgiDAV configuration directory:'''</p></li>
</ol><pre>sudo mkdir -p /etc/wsgidav</pre>
<ol  start="3" style="list-style-type: decimal;"><li><p>'''Generate an SSL certificate for WsgiDAV:'''</p></li></ol>

<pre> sudo openssl req -x509 -nodes -days 365 -newkey rsa:2048 -keyout /etc/ssl/private/wsgidav.key -out /etc/ssl/certs/wsgidav.crt</pre>
This creates a self-signed SSL certificate. In a production environment, use a certificate from a trusted Certificate Authority. When having localhost connect to localhost in your closet… this will do.

<ol start="4" style="list-style-type: decimal;">
<li>'''Create and edit the WsgiDAV configuration file:'''</li></ol>

<pre>sudo nano /etc/wsgidav/wsgidav.yaml</pre>
<ol start="5" style="list-style-type: decimal;">
<li>'''Add the following content to the configuration file, editing <code>/home/louis/webdavroot</code> with the directory you will use for documents:'''</li></ol>

<pre>  host: 0.0.0.0
  port: 8080
  ssl_certificate: /etc/ssl/certs/wsgidav.crt
  ssl_private_key: /etc/ssl/private/wsgidav.key
  enable_https: true
  
  
  fs_dav_provider:
      follow_symlinks: true
  
  `provider_mapping:
      '/webdav': '/home/louis/webdavroot'
  
  
  http_authenticator:
    domain_controller: wsgidav.dc.pam_dc.PAMDomainController
    accept_basic: true
    accept_digest: false
    default_to_digest: false
  
  pam_dc:
    service: &quot;login&quot;
    allow_users: &quot;all&quot;
  
  verbose: 3
  
  property_manager: true
  lock_storage: true
  
  middleware_stack:
    - wsgidav.error_printer.ErrorPrinter
    - wsgidav.http_authenticator.HTTPAuthenticator
    - wsgidav.dir_browser.WsgiDavDirBrowser
    - wsgidav.request_resolver.RequestResolver
  
  dir_browser:
    enable: true
    icon: true
    response_trailer: true</pre>
This configuration sets up SSL, defines shared directories, and configures authentication.

<ol start="7" style="list-style-type: decimal;">
<li><p>'''Add the following content to the service file:'''</p>
<pre>[Unit]
Description=WsgiDAV WebDAV Server
After=network.target

[Service]
ExecStart=/usr/local/bin/wsgidav --config=/etc/wsgidav/wsgidav.yaml
Restart=always

[Install]
WantedBy=multi-user.target</pre>
<p>This creates a systemd service for automatically starting WsgiDAV.</p></li>
<li><p>'''Set correct permissions for the configuration file:'''</p>
<pre>sudo chown root:root /etc/wsgidav/wsgidav.yaml
sudo chmod 644 /etc/wsgidav/wsgidav.yaml</pre>
<p>This make sures only root can modify the configuration file.</p></li>
<li><p>'''Enable and start the WsgiDAV service:'''</p>
<pre>sudo systemctl enable wsgidav.service
sudo systemctl start wsgidav.service</pre>
<p>This enables the service to start on boot and starts it immediately.</p></li></ol>

Now, it’s time to go back to the onlyoffice window we were at before to enter the WebDAV server information. '''See how mine is /webdav? That’s because'''

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:lu55028jxemg_tmp_77f5a739.png
</gallery>

<span id="understanding-file-locations"></span>

==== 3.2 Understanding file locations ====

These lines in the WsgiDAV configuration file are responsible for setting the directory that onlyoffice will see on our system. Obviously, if your name is not louis, yours will be different. Edit it accordingly.

<code>provider_mapping: '/webdav': '/home/louis/webdavroot'</code>

'''Remember, WsgiDAV will only let me have one directory that I can get into when I start it up. The way I got around this was as follows, so that my Documents directory and my androidbackup directories would both be visible by onlyoffice:'''

<pre>ln -s /home/louis/Documents /home/louis/webdavroot
ln -s /home/louis/androidstuff /home/louis/webdavroot</pre>
Now, my '''Documents''' folder in my home directory as well as my '''androidstuff''' syncthing backup directory with all of my phone’s files will be viewable by onlyoffice!

<span id="configure-firewall-ufw"></span>
==== 3.3 Configure Firewall (UFW) ====

UFW (Uncomplicated Firewall) provides a user-friendly interface for managing iptables. There is no need for anything besides onlyoffice to ever contact our WebDAV server, so we are going to make sure only localhost can contact our WebDAV server.

If you think this is ridiculous, it is. Onlyoffice needs to let me access files on my local server that are already there.

<ol style="list-style-type: decimal;">
<li><p>'''Allow all outgoing traffic:'''</p>
<pre>sudo ufw default allow outgoing</pre></li>
<li><p>'''Allow incoming traffic on port 8080 from specific sources:'''</p>
<pre>sudo ufw allow from 192.168.5.5 to any port 8080 proto tcp
sudo ufw allow from 127.0.0.1 to any port 8080 proto tcp
sudo ufw allow from 172.17.0.0/16 to any port 8080 proto tcp
sudo ufw allow from 172.18.0.0/16 to any port 8080 proto tcp</pre>
<p>This allows HTTPS traffic to WsgiDAV only from specific IP ranges.</p></li>
<li><p>'''Enable the firewall:'''</p>
<pre>sudo ufw enable</pre>
<p>This activates the firewall with the configured rules.</p></li></ol>

<span id="step-4-make-sure-this-works"></span>
== Step 4: Make sure this works ==

* Open onlyoffice, and try to open files

<div class="figure">

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:lu55028jxemg_tmp_1b047a4c.png
</gallery>

</div>
<div class="figure">

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:lu55028jxemg_tmp_6153f9b3.png
</gallery>

</div>
<div class="figure">

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:lu55028jxemg_tmp_dab77d16.png
</gallery>

</div>
<div class="figure">

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:lu55028jxemg_tmp_f9684b3a.png
</gallery>

</div>
<div class="figure">

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:lu55028jxemg_tmp_56592c9d.png
</gallery>

</div>
<div class="figure">

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:lu55028jxemg_tmp_5931a5e.png
</gallery>

</div>
<div class="figure">

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:lu55028jxemg_tmp_38c777ab.png
</gallery>

</div>
<div class="figure">

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:lu55028jxemg_tmp_5925a9a4.png
</gallery>

</div>
<div class="figure">

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:lu55028jxemg_tmp_148a1456.png
</gallery>

</div>
<span id="step-5-optional-set-up-email-in-onlyoffice"></span>
== Step 5 (optional): set up email in onlyoffice ==

<span id="viewing-email-right-in-the-web-browser"></span>
=== Viewing email right in the web browser ===

If you set up onlyoffice as an email client for your mailcow server, you can view your email within onlyoffice. This means you can open documents directly within onlyoffice within the browser tab where you have your email loaded. Very nice!

<span id="freepbx-and-unitel-sip-trunking-setup"></span>
= FreePBX and UniTel SIP Trunking Setup =

<span id="introduction"></span>
== Introduction ==

Just like with self managed mail; this will be high maintenance, low reward, and is a very bad idea - like anything worth doing. This guide provides detailed instructions on setting up a '''FreePBX''' system with '''UniTel SIP Trunking'''.

<span id="why-customize-your-phone-system"></span>
== Why Customize Your Phone System? ==

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:lu67917r1ezu_tmp_7fc7682.png
</gallery>

<span id="callerid-hacks-to-make-calls-go-faster"></span>
=== CallerID hacks to make calls go faster ===

One of the fun things you can do with this setup is integrate it with your customer relationship management software like I did with repairshopr. So, instead of the usual caller ID, you can have the status of a customer’s ticket show up. Back when I was the only one doing repairs at my store, this was a lifesaver. Most calls were simple status checks, and I could handle them while soldering, thanks to a Bluetooth headset.

* '''Caller ID Customization''': Instead of just a name, I saw ticket status in the caller ID too!
* '''Efficiency''': I could handle calls without stopping my work!
* '''Customer Satisfaction''': Instant info made customers feel like you know them better than they know themselves.

<span id="automatically-send-mean-customers-to-an-extension-where-allison-smith-tells-them-to-go-fuck-themselves-d"></span>
=== Automatically send mean customers to an extension where Allison Smith tells them to go fuck themselves :D ===

Rossmann Repair has never made use of this feature.

<span id="make-telemarketers-miserable-by-installing-a-program-that-messes-with-them-lenny"></span>
=== Make telemarketers miserable by installing a program that messes with them: Lenny ===

The customization possibilities are endless, and that’s what makes this so much fun. Now, let’s get into how to build your own system.

<span id="step-1-preparing-a-freepbx-installation"></span>
== Step 1: Preparing a FreePBX installation ==

<span id="step-1-set-up-a-freepbx-virtual-machine"></span>
=== Step 1: Set up a FreePBX virtual machine ===

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:lu67917r1ezu_tmp_bfad3d9b.png
</gallery>

<span id="download-debian-12-iso"></span>
==== 1.1 Download Debian 12 ISO ====

You used to download FreePBX as its own distro, which was based on CentOS. They switched to Debian after some recent CentOS/Red Hat controversy.

<ol style="list-style-type: decimal;">
<li><p>Open a terminal window or use a web browser within your '''happycloud''' server that is running '''Virtual Machine Manager''' to host all of your virtual machines. In our case, that’s '''192.168.5.2'''.</p></li>
<li><p>'''Download and install''' [https://www.debian.org/CD/ ''Debian 12''] on the machine designated for FreePBX.</p>
<pre>wget https://cdimage.debian.org/debian-cd/current/amd64/iso-cd/debian-12.0.0-amd64-netinst.iso -P ~/Downloads</pre></li>
<li><p>Make sure the download completes successfully.</p></li></ol>

<span id="move-the-debian-iso-to-the-correct-directory"></span>
==== 1.2 Move the Debian ISO to the Correct Directory ====

<ol style="list-style-type: decimal;">
<li><p>Move the downloaded ISO to <code>/var/lib/libvirt/images</code>:</p>
<pre>sudo mv ~/Downloads/debian-12.0.0-amd64-netinst.iso /var/lib/libvirt/images/</pre></li>
<li><p>Set the correct permissions and ownership for the ISO:</p>
<pre>sudo chmod 644 /var/lib/libvirt/images/debian-12.0.0-amd64-netinst.iso
sudo chown libvirt-qemu:kvm /var/lib/libvirt/images/debian-12.0.0-amd64-netinst.iso</pre>
<p><gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:lu67917r1ezu_tmp_b51c10bd.png
</gallery></p></li></ol>

<span id="launch-virtual-machine-manager"></span>
==== 1.3: Launch Virtual Machine Manager ====

Open Virtual Machine Manager from the Openbox menu by right-clicking the desktop, going to system, and then running virtual machine manager. Or run:

<pre>virt-manager</pre>
<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:lu67917r1ezu_tmp_151745e3.png
</gallery>

<span id="create-a-new-virtual-machine"></span>
==== 1.4 Create a New Virtual Machine ====

# Click '''Create a new virtual machine'''.
# Select '''Local install media (ISO image or CDROM)''' and click '''Forward'''.
# Click '''Browse…''' and navigate to <code>/var/lib/libvirt/images/</code> to select <code>debian-12.0.0-amd64-netinst.iso</code>.
# Choose '''Detect automatically''' for the OS type or manually set it as '''Debian 12'''.
# Click '''Forward'''.

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:lu67917r1ezu_tmp_4bc43bb4.png
File:lu67917r1ezu_tmp_ba3e10c4.png
</gallery>

<span id="configure-vm-resources"></span>
==== 1.5: Configure VM Resources ====

# '''Memory &amp; CPU:'''
#* Assign '''4096 MB''' of RAM (or more, but the idea of giving more than 4 gigs to a phone system hurts me).
#* Assign '''2 CPUs''' (adjust based on available resources).
# '''Storage:'''
#* Select '''Create a disk image for the virtual machine'''.
# Allocate at least '''20 GB''' for storage. Choose more if you expect larger usage.
# Click '''Forward'''.

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:lu67917r1ezu_tmp_47157042.png
</gallery>

<span id="set-up-networking"></span>
==== 1.6 Set Up Networking ====

* Make sure the network selection is set to '''Bridge''' and matches your LAN network (e.g., <code>br0</code>). This will allow your VM to get a static IP from your existing network. Under '''Network''' settings, make sure it’s set to '''Bridge mode''' for proper network integration.

<span id="set-up-freepbx-to-start-on-boot"></span>
==== 1.7 Set up FreePBX to start on boot ====

<pre>virsh autostart freepbx</pre>
* Check that this is set up properly by typing <code>virsh dominfo freepbx</code> and seeing if the autostart line is set to enable.
* If you don’t do this, you will realize once it is too late and you’ve left your house after you have rebooted your server (for whatever reason) that your phone system is dead. Don’t do that.

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:lu67917r1ezu_tmp_e22151da.png
File:lu67917r1ezu_tmp_299cdef5.png
File:lu67917r1ezu_tmp_331573f2.png
File:lu67917r1ezu_tmp_c164c52b.png
File:lu67917r1ezu_tmp_474da8d6.png
File:lu67917r1ezu_tmp_b401b780.png
File:lu67917r1ezu_tmp_27a9fd11.png
File:lu67917r1ezu_tmp_f48a1f82.png
File:lu67917r1ezu_tmp_f01c615f.png
File:lu67917r1ezu_tmp_daa809f6.png
File:lu67917r1ezu_tmp_3938eead.png
File:lu67917r1ezu_tmp_863c1219.png
File:lu67917r1ezu_tmp_989c8592.png
File:lu67917r1ezu_tmp_e4c7555a.png
File:lu67917r1ezu_tmp_12e25dc6.png
File:lu67917r1ezu_tmp_362ff89f.png
File:lu67917r1ezu_tmp_82b7c46.png
File:lu67917r1ezu_tmp_fb20c8a5.png
File:lu67917r1ezu_tmp_c84cc7cd.png
File:lu67917r1ezu_tmp_e0ce50d3.png
File:lu67917r1ezu_tmp_40edf977.png
File:lu67917r1ezu_tmp_53383ce3.png
File:lu67917r1ezu_tmp_e0e659a5.png
File:lu67917r1ezu_tmp_d7ccc18d.png
File:lu67917r1ezu_tmp_d5c2b085.png
</gallery>

<span id="debian-12-installation-setup"></span>
==== 1.8 Debian 12 Installation Setup ====

# Follow the Debian installer prompts:
#* '''Language''': Choose your preferred language.
#* '''Location''': Set your country.
#* '''Keyboard''': Select your preferred layout.
#* '''Hostname''': Set the hostname as <code>freepbx</code>.
# '''Domain Name''':
#* You can leave this blank.
# '''Set the Root Password''':
#* Choose a secure password and confirm it.
# '''Create a New User''':
#* Add a user. I added a user named <code>louis</code> for myself.
# '''Partitioning''':
#* Choose '''“Guided - use entire disk and set up LVM”'''.
#* DO NOT USE ENCRYPTION - REMEMBER, THE HOST SYSTEM THIS IMAGE IS ON IS ALREADY AN ENCRYPTED DISK!!
#* Select the disk and proceed.
#* Confirm changes to write the partitions.
#* The disk device will most likely be something like <code>/dev/vda</code>.

<span id="post-installation-configuration-test"></span>
==== 1.9 Post-Installation Configuration Test ====

After rebooting, log in as root or your user.

Make sure network connectivity works:

<pre>ping 8.8.8.8
hostnamectl</pre>
<span id="step-2-preparing-debian-12-for-freepbx-installation"></span>
=== Step 2: Preparing Debian 12 for FreePBX Installation ===

This guide provides instructions on performing basic maintenance on a fresh Debian 12 installation and then downloading and running the FreePBX installation script. Follow the steps carefully to ensure a smooth setup.

<span id="configure-network-settings"></span>
==== 2.1 Configure Network Settings ====

# Log in with your username and password on the virt-manager screen on your host computer (the one hosting all the virtual machines).
# Type <code>ip addr show</code> and find which interface shows your IP address.
#* Remember its name for later.
#* It should be something like <code>enp1s0</code>.
# Become root:

<pre>su</pre>
<ol start="4" style="list-style-type: decimal;">
<li><p>Make a network configuration file like this:</p>
<blockquote><p>Use the name of your network interface in place of <code>enp1s0</code>.</p></blockquote></li></ol>

<pre>nano -w /etc/systemd/network/enp1s0.network</pre>
<pre>[Match]
Name=enp1s0 #put name of your network interface in place of enp1s0

[Network]
Address=192.168.5.6/24
Gateway=192.168.5.1
DNS=192.168.5.1</pre>
<ol start="5" style="list-style-type: decimal;">
<li>Hit <code>ctrl-x</code>, then <code>y</code> to save.</li></ol>

<pre>systemctl restart systemd-networkd</pre>
<ol start="6" style="list-style-type: decimal;">
<li><p>Make sure your IP address has changed to a static IP by typing <code>ip addr show</code> and checking.</p>
<ul>
<li><p>'''Static IP''': Set the IP address to 192.168.5.6.</p></li>
<li><p>'''Gateway''': Use 192.168.5.1.</p></li>
<li><p>'''DNS Server''': Set to 192.168.5.1.</p></li></ul>
</li></ol>

<span id="do-basic-maintenance-on-debian-12"></span>
==== 2.2 Do Basic Maintenance on Debian 12 ====

<ul>
<li><p>'''Update Package Lists'''</p>
<ul>
<li><p>Refresh the package lists to make sure you get the latest versions</p></li>
<li><p>Upgrade packages</p></li>
<li><p>Remove junk, all with the following line:</p></li></ul>

<pre>sudo apt update ; sudo apt upgrade -y ; sudo apt autoremove -y</pre></li></ul>

<span id="download-run-freepbx-install-script"></span>
==== 2.3 Download &amp; run ''[https://github.com/FreePBX/sng_freepbx_debian_install FreePBX install script]'' ====

<ol style="list-style-type: decimal;">
<li><p><code>ssh</code> to the Debian system as <code>louis</code></p></li>
<li><p>Check ''[https://github.com/FreePBX/sng_freepbx_debian_install FreePBX page]'' for the latest script since the URL will change over time.</p></li>
<li><p>Download the file using <code>wget</code>:</p>
<pre>su
wget https://github.com/FreePBX/sng_freepbx_debian_install/raw/master/sng_freepbx_debian_install.sh -O /tmp/sng_freepbx_debian_install.sh</pre></li>
<li><p>Make the script executable</p></li></ol>

<pre>   su # become root user
   chmod +x /tmp/sng_freepbx_debian_install.sh</pre>
<ol start="5" style="list-style-type: decimal;">
<li><p>Run the FreePBX Installation Script</p>
<pre>bash /tmp/sng_freepbx_debian_install.sh</pre>
<ul>
<li>The script will handle the setup of Asterisk, Apache, MySQL, Postfix, etc., all necessary FreePBX modules.</li></ul>
</li></ol>

<span id="step-3-visit-freepbx-web-interface"></span>
=== Step 3: Visit FreePBX Web Interface ===

# '''Open a Web Browser'''
#* Use a browser on a device connected to the same network, or a device that is connected via OpenVPN. Remember, none of this is open to the public!
# '''Navigate to the FreePBX IP'''
#* Access FreePBX by entering the following URL: ''http://192.168.5.6/admin''
#* If you used a different IP configuration than I did, enter that IP.
# '''Answer prompts for user/password'''
#* Follow the setup wizard to configure your admin user, language settings, &amp; other preferences. But don’t get started messing around with anything serious just yet.

<span id="step-3-configuring-unitel-as-phone-service-provider"></span>
== Step 3: Configuring UniTel as Phone Service Provider ==

Now that we have a working FreePBX installation, we’re ready to set things up with an SIP trunk provider, aka ''“the phone company”''. I use UniTel for this at the UniTel Customer Portal at ''unitelcustomer.com''.

<span id="setting-up-sip-trunking"></span>
=== Setting up SIP trunking ===

A SIP trunk account is like your phone’s connection to the outside world. Think of it as paying for internet but for your phone calls. Without it, your PBX system is just an intercom for internal calls. You need a SIP trunk to make and receive calls from the outside world.

<span id="choosing-an-sip-trunk-provider"></span>
=== Choosing an SIP Trunk Provider ===

I recommend '''Unitel''' for SIP trunking. They’re solid, reliable, and unlike some other providers, they don’t just resell AWS in the cheapest way possible. I used to use VoicePulse, but they were terrible and went out of business a few years ago. They had no redundancy, so if some single AWS instance went down, you were out of luck.

<span id="john-grossbard-studio-landlord-seinfeld-character"></span>
=== John Grossbard: Studio Landlord &amp; Seinfeld Character ===

There was an episode of Seinfeld where he said ''“I GOT GROSSBAR’D!”'' Supposedly this was a reference to a minor argument between him and Larry David.

John Grossbard was the owner of Planet to Planet Studios when I rented a space from him for my screen wholesaling company in the basement of 251 W. 30th St in NYC, back when this was called the “Music Building,” before it was remodeled to appeal to hipster-0%-interest-rate-funded-fad-yuppie-startups.

I was here because I had no credit and he didn’t ask for a security deposit. When a friend of mine brought up an issue of bedbugs, he looked at us and said ''“If I made this place any nicer, you couldn’t afford it.”''

He wasn’t wrong. This stuck with me my entire adult life.

Unitel has two websites. One is their main website, and one is a website that looks like some 1997 Slashdot site. The website with the 1997 Slashdot look is what we want. It’s not too nice – that means we can ''“afford”'' it.

<span id="the-benefits-of-your-own-pbx-revisited"></span>
=== The Benefits of Your Own PBX; Revisited ===

In all seriousness, one of the benefits of hosting our own PBX is that we '''DON’T''' pay by the user. We '''DON’T''' pay by the feature. All we pay for is SIP trunking—any features &amp; functionality are added by '''US''', to '''OUR''' PBX, that we control.

We don’t need them to make it any nicer. If they did, you couldn’t afford it.

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:lu67917r1ezu_tmp_df8469aa.png
File:lu67917r1ezu_tmp_a1bc014.png
File:lu67917r1ezu_tmp_ea4afdcf.png
File:lu67917r1ezu_tmp_9c0b5e5b.png
</gallery>

<blockquote>'''HINT:''' Avoid the fancy, infinite-scrolly websites like UnitelPhone.com. Instead, go for the classic UnitelCustomer.com. If it looks like it’s from 1997, you’re in the right place! It’s straightforward and gets the job done.
</blockquote>
Having your own PBX means no more paying per user. You pay based on trunk usage, not the number of users. Some providers nickel and dime you on the number of extensions you have, visual voicemail, call recording, etc. With SIP trunking, they have no idea what is going on, so they can’t bill you by-the-extension or by-the-feature. It’s like a VPN for your calls—they don’t know how many extensions you have or if you’re recording calls. All they know is the call came in or went out. No extra charges for features like visual voicemail, lenny, call recording, or the voice of Allison Smith telling callers to go fuck themselves; which is the reason I set this up to begin with 14 years ago. :D

<span id="register-for-a-unitel-account"></span>
==== 3.1 Register for a Unitel Account ====

* Open a web browser and navigate to: [https://signup.unitelgroup.com/sip-trunking-plans https://signup.unitelgroup.com/sip-trunking-plans]
* Create an account with the nice SIP trunking people

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:lu67917r1ezu_tmp_be593c35.png
File:lu67917r1ezu_tmp_3fa0de28.png
File:lu67917r1ezu_tmp_6a3ae870.png
</gallery>

<span id="set-up-an-endpoint-this-is-where-inbound-calls-get-sent-to"></span>

==== 3.2 Set Up an Endpoint (This is Where Inbound Calls Get Sent To) ====

Endpoints are where your call is sent when a call comes in on a number you have in Unitel. When you create a phone number in Unitel, it will ask where you want to send calls that come into that number. We’re going to set up the endpoint first so when we create a number, we’ll already have an endpoint to send it to.

'''Navigate to the “Endpoints” Section'''

# From the main dashboard, go to '''“Settings”'''.
# Click on '''“Endpoints”'''.
# '''Create a New Endpoint'''
# Click on '''“Add Endpoint”'''.
#* Fill in the following details:
#** '''Endpoint Description''': Enter a name that describes the endpoint (e.g., closet pbx).
#** '''Endpoint Destination''': Insert the dynamic DNS entry (e.g.<code>louishomeserver.chickenkiller.com</code>) that you set up back in the FreeDNS Dynamic DNS section of this guide. This should resolve to your PBX’s external IP address. When a call comes in on a specific number, it is going to send the call to your PBX at this IP.
# Click '''“Add Endpoint”''' to complete the setup.

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:lu67917r1ezu_tmp_8b8d5b23.png
File:lu67917r1ezu_tmp_70f53605.png
File:lu67917r1ezu_tmp_70f56fbb.png
File:lu67917r1ezu_tmp_ae6d3290.png
File:lu67917r1ezu_tmp_16e56098.png
</gallery>

<span id="get-configure-phone-numbers"></span>
==== 3.3 Get &amp; Configure Phone Numbers ====

# '''Navigate to the “Numbers” Section'''
#* From the main dashboard, after clicking on '''“Numbers”''', click on '''“Add Number”'''.
#* Buy a number.
# '''Assign the Purchased Number to the Endpoint'''
#* After purchasing, go to '''“Manage Numbers”'''.
#* Find the purchased number and click the dropdown under '''“Actions”''' and click '''“Number Mode”'''.
#* Select '''“Forward to Endpoint”''': Select the endpoint you created earlier (e.g., closet pbx).
#* Click '''Update'''.

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:lu67917r1ezu_tmp_351530c2.png
File:lu67917r1ezu_tmp_5d30f03a.png
File:lu67917r1ezu_tmp_5eb714e2.png
</gallery>

<span id="add-a-trunk-in-unitel"></span>
==== 3.4 Add a Trunk in Unitel ====

'''Purpose of an SIP trunk:'''

An SIP trunk is what attaches you to the world, similar to how your cable modem &amp; spectrum or verizon connects you to the rest of the internet. A '''trunk''' is a connection between your phone system (PBX) and the external phone network. It allows your system to make and receive calls to/from the outside world. Setting up a trunk in Unitel is necessary.

The purpose of the trunk is to provide a pathway for your PBX to route calls to and from the public telephone network. Without a properly configured trunk, your system won’t be able to communicate with external phone numbers. Which is what I have been doing for three weeks while writing this guide.

# '''Log in to the Unitel Admin Interface'''
#* Open your web browser and go to the Unitel admin interface.
#* Log in using your credentials.
# '''Step 2: Add a New Trunk'''
#* Navigate to Manage SIP Trunks.
#* Click on Add Trunk.
# '''Step 3: Configure Trunk Details'''
#* Trunk Description: Enter a descriptive name for your trunk (e.g., Main Trunk).
#* Trunk Type: Select '''General use/Conversational'''.
# Click Add Trunk to save the new trunk.
# Click Apply Config to activate the trunk.
# '''NOTE YOUR CREDENTIALS!'''
#* During the trunk setup, you will be provided with a username and password. Be sure to note your username &amp; password in a password manager of some sort as it will be needed later when configuring the trunk in your PBX system.

<span id="step-4-setting-up-freepbx-with-unitel-phone-service"></span>
== Step 4: Setting up FreePBX with Unitel phone service ==

Now that your phone service provider is set up, we can configure <code>freepbx</code> to connect to it &amp; receive &amp; send phone calls. We’ll be using UniTel credentials to sign into our trunk.

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:lu67917r1ezu_tmp_2708b4e9.png
File:lu67917r1ezu_tmp_dcdd2d35.png
File:lu67917r1ezu_tmp_dcb5bd91.png
File:lu67917r1ezu_tmp_cfef6f4a.png
File:lu67917r1ezu_tmp_d36e7da6.png
File:lu67917r1ezu_tmp_2df4038d.png
File:lu67917r1ezu_tmp_88afd9c8.png
File:lu67917r1ezu_tmp_f1c06a36.png
File:lu67917r1ezu_tmp_7fdbd089.png
File:lu67917r1ezu_tmp_9a45904f.png
File:lu67917r1ezu_tmp_bfc2022b.png
</gallery> Get into FreePBX interface:

<ul>
<li><p>Open a web browser and navigate to:</p>
<p><code>http://freepbx.home.arpa/</code> or <code>http://192.168.5.5/</code></p></li>
<li><p>Log in</p></li></ul>

<span id="add-a-new-sip-trunk"></span>
==== 4.1 Add a New SIP Trunk ====

# '''Navigate to the Trunk Configuration'''
# Go to '''Connectivity''' &gt; '''Trunks'''.
# Click '''Add Trunk''' and choose '''Add SIP (chan_pjsip) Trunk'''.
# '''Configure the General Settings'''
#* '''Trunk Name''': Enter a happy name, like UniTel_SIP.
#* '''Hide CallerID''': Set to No.
#* '''Outbound CallerID''': Enter your UniTel DID (your phone number) in e.164 format (e.g., 13475522258 for rossmann repair group).
#* '''CID Options''': Choose Allow Any CID.
#* '''Dial Number Manipulation Rules'''
#* '''Outbound Dial Prefix''': Make sure all outgoing calls use the 11-digit e.164 format (e.g., 1NXXNXXXXXX).
#* Add a rule if needed to prepend 1 for local or long-distance numbers:
#** Match Pattern: NXXNXXXXXX
#** Prepend: 1

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:lu67917r1ezu_tmp_8b0dd42e.png
File:lu67917r1ezu_tmp_dcdd4148.png
File:lu67917r1ezu_tmp_f5e688e2.png
File:lu67917r1ezu_tmp_cb4b6061.png
File:lu67917r1ezu_tmp_46f1721e.png
File:lu67917r1ezu_tmp_c63f74a0.png
File:lu67917r1ezu_tmp_77abb232.png
</gallery>

<span id="pjsip-settings-configuration-in-trunk-configuration"></span>
==== 4.2 PJSIP Settings Configuration in trunk configuration ====

# '''Go to the “PJSIP Settings” Tab'''
#* '''Username''': Enter the SIP username provided by UniTel.
#* '''Secret''': Enter the SIP password from UniTel.
#* '''SIP Server (SIP Host)''': Set to <code>sip.unitelgroup.com</code>, '''this may change over time. Make sure you check Unitel’s instructions that they offer on unitelcustomer.com after you log in.''' They’re nice people &amp; provide all this for you in plain English.
#* '''Authentication''': Set to Outbound, should be checked by default.
#* '''Registration''': Choose Send, should be checked by default.
# '''Advanced Options'''
#* '''From Domain''': Enter <code>sip.unitelgroup.com</code>.
#* '''Context''': Use <code>from-pstn-toheader</code> (This allows FreePBX to correctly handle incoming SIP headers from UniTel). This should be set by default.
# Go to '''“Codecs” tab'''
#* Check the [https://www.unitelcustomer.com/sip-getting-started-guide codecs that Unitel allows]. <gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:lu67917r1ezu_tmp_e238d46d.png
</gallery>
#** G723
#** G729
#** G711
#*** G711 is actually '''ulaw''' and '''alaw''' in the list.
#* This may change over time, check [https://www.unitelcustomer.com/sip-getting-started-guide Unitel’s page] for details.
#* Unitel are nice people &amp; want you to be successful in setting up your PBX. They provide you with all of this information.
# '''Submit and Apply'''
#* Click '''Submit'''.
#* Click '''Apply Config''' on the top-right to save changes and reload the configuration.

<span id="step-5-internal-call-handling-and-call-routing-setup"></span>
== Step 5: Internal Call Handling and Call Routing Setup ==

Now we start the process of setting up internal call handling by creating extensions and ring groups, as well as defining call routing to manage inbound and outbound calls using the UniTel SIP trunk in FreePBX 17. Having a trunk is useless if we don’t have any phones set up.

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:lu67917r1ezu_tmp_ce66bf80.png
File:lu67917r1ezu_tmp_9780cee6.png
File:lu67917r1ezu_tmp_de1c9c6c.png
File:lu67917r1ezu_tmp_cbd71b4b.png
File:lu67917r1ezu_tmp_ece5f184.png
File:lu67917r1ezu_tmp_ba101465.png
</gallery> <gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:lu67917r1ezu_tmp_20ad80b1.png
</gallery> 5.1 Create an Extension

Extensions are individual phones. Alice has an extension for a phone on her desk, Mark has an extension for a phone on his desk, and so on and so forth. Each extension has a number for internal calls. Your desk phone could be 101 - this means people inside FreePBX connected directly to your PBX can call 101. This needs to be done first.

* '''Log in to the FreePBX Admin Interface'''
* '''Navigate to Extensions'''
** Go to '''Connectivity''' &gt; '''Extensions'''.
** Click '''Add Extension'''.
** Choose '''Add SIP (chan_pjsip) Extension'''.
* '''Configure the Extension'''
** '''User Extension''': Enter a unique extension number (e.g., 101).
** '''Display Name''': Enter the name for this extension (e.g., Office Phone).
** '''Secret''': Enter a strong password for the extension or let FreePBX generate one automatically. It’s a good idea to add this to your password manager like Bitwarden so you have it later. Don’t put this on a post-it note.
* '''Voicemail''': Enable if you want voicemail for this extension.
** Set email address to the email you want voicemail sent to.
** Set Voicemail password to the password you want to have to dial to access voicemail (we will never use this archaic method, we will get voicemails emailed to us).
* In '''Advanced''' you can set up call recording.
* '''Submit and Apply Changes'''

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:lu67917r1ezu_tmp_e706e9c9.png
File:lu67917r1ezu_tmp_82fe25e7.png
File:lu67917r1ezu_tmp_a8632a6a.png
</gallery> 5.2 Configure Ring Groups

When someone calls 3475522258 for my business, I don’t want one phone to ring. I want all of them to ring. This is what ring groups are for. We create one number that rings a bunch of different phones.

# Go to '''Applications''' &gt; '''Ring Groups'''.
# Click '''“Add Ring Group”'''.
#* '''Ring Group Number''': Enter a unique number for the ring group (e.g., 600).
#* '''Group Description''': Enter a name for this ring group (e.g., Office Ring Group).
#* '''Ring Strategy''': Choose how calls should be distributed (e.g., Ringall to ring all devices simultaneously). '''Ringall is what you want 99% of the time. Use ringall if you are confused.'''
#* '''Extension List''': Add the extensions you want to include in this ring group (e.g., 101). Everyone here will have their phone ring when this ring group is called. In an office with one phone number, you would want to put every extension here of the people you want to pick up the phone when a customer calls.
# Customize settings like '''Ring Time''', '''Destination if No Answer''', and '''Call Recording'''. I usually set this to the voicemail of a particular extension.
# Click '''“Submit”'''.
# Click '''“Apply Config”''' to activate the ring group.
# '''REMEMBER, YOU NEED TO SET A DESTINATION IF NO ANSWER SO THAT PEOPLE CAN LEAVE VOICEMAILS.'''
#* Mitt Romney famously said ''[https://www.youtube.com/watch?v=KlPQkd_AA6c&t=4s “ring groups are people, my friend.”]''
#* They are not! ''People have voicemails – ring groups do not.''

<span id="step-6-define-call-routing"></span>
== Step 6: Define Call Routing ==

Inbound routes define what we do when someone calls a particular phone number.

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:lu67917r1ezu_tmp_c7b43510.png
File:lu67917r1ezu_tmp_97e7fd79.png
File:lu67917r1ezu_tmp_df1b9161.png
</gallery>

<span id="set-up-call-flow-control"></span>
==== 6.1 Set Up Call Flow Control ====

Call flow control allows you to change where calls go by dialing a number on your phone.

For instance, let’s say your business hours are 11 AM to 7 PM. You can set it up so that when you close, you dial <code>*2886</code> on your phone to send the calls directly to voicemail. Then, when you open the next day, you dial <code>*2886</code> again and your calls switch back to going to all of your business phones rather than go to voicemail.

I like this more than I like call scheduling because I set it manually. If I come to work early, I may want to answer the phone early. If I stay late, I may want to answer the phone late.

Rather than set up my phone number to go straight to my ring group, I set it up to go to call flow control. Then, I set up call flow control to go to my ring group, and my ring group to go to my extensions.

# '''Navigate to Inbound Routes'''
# Go to '''“Applications”''' &gt; '''“Call Flow Control”'''.
# Click '''“+ Add”'''.
# '''Configure the Call Flow Control'''
#* '''Call Flow Toggle Feature Code Index''': This just means what number you enter into your phone to change it. Whatever you add here will be ''in front'' of a 28. So, if you enter 86, that means dialing <code>*2886</code> on your phone will toggle where your calls go.
#** On older phones like the Cisco SPA525G, this doesn’t work since they seem to only support ''two digits'' in front of a <code>*</code> rather than four. :(
#* '''Description''': Describe what the point of this is so you know for later.
#* '''Current Mode''': This sets how calls are going when you initially finish setting this up.
#* '''Normal Flow''': This sets where calls go before you toggle call flow control. Put the default here. For me, that’s ringing all of my office phones at the '''Ring Group''' I set up earlier. Enter '''Ring Groups''' and then put the '''Ring Group''' you created here.
#* '''Override Flow''': This is where calls will go when you dial <code>*2886</code> and toggle this feature on. Set this to '''Voicemail''' and then the voicemail of the extension we created.
#* '''CallerID Number''': Same as the DID number.
#* '''Description''': Provide a description for this route (e.g., Rossmann repair business number).
#* '''Set Destination''': Choose '''“Ring Groups”''' and select the ring group number you created earlier (e.g., 600 - Office Ring Group).
# '''Submit and Apply Changes'''

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:lu67917r1ezu_tmp_2d63ee.png
File:lu67917r1ezu_tmp_2457af08.png
File:lu67917r1ezu_tmp_cd27e90.png
</gallery>

<span id="set-up-an-inbound-route"></span>
==== 6.2 Set Up an Inbound Route ====

# '''Navigate to Inbound Routes'''
# Go to '''“Connectivity”''' &gt; '''“Inbound Routes”'''.
# Click '''“+ Add Inbound Route”'''.
# '''Configure the Inbound Route'''
#* '''DID Number''': Enter your UniTel DID in e.164 format (e.g., 13475522258). ''Put a 1 in front of your number in the US!''
#* '''CallerID Number''': Same as the DID number.
#* '''Description''': Provide a description for this route (e.g., rossmann repair business number).
#* '''Set Destination''': Choose '''“Ring Groups”''' and select the ring group number you created earlier (e.g., 600 - Office Ring Group).
# '''Submit and Apply Changes'''

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:lu67917r1ezu_tmp_1602ae25.png
File:lu67917r1ezu_tmp_69cfd1f3.png
File:lu67917r1ezu_tmp_5fce659.png
</gallery>

<span id="set-up-callerid"></span>
==== 6.3 Set Up CallerID ====

# '''Navigate to CID Superfecta'''
#* Go to '''“Admin”''' &gt; '''“CID Superfecta”'''.
#* Click '''Yes''' on the callerID sources you think would be useful.
# '''Navigate to Inbound Routes'''
#* Go to '''Connectivity''' &gt; '''Inbound Routes'''.
#* Go to the '''Other''' tab.
#* Choose '''CID Lookup Source''' as '''CID Superfecta'''.
# '''Submit and Apply Changes'''

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:lu67917r1ezu_tmp_e50a92c9.png
File:lu67917r1ezu_tmp_94ce168a.png
File:lu67917r1ezu_tmp_3871242b.png
File:lu67917r1ezu_tmp_a057d15a.png
File:lu67917r1ezu_tmp_704ad695.png
File:lu67917r1ezu_tmp_65faf5c2.png
File:lu67917r1ezu_tmp_755f860.png
</gallery>

<span id="configure-outbound-route-for-making-calls"></span>
==== 6.4 Configure Outbound Route for Making Calls ====

# '''Navigate to Outbound Routes'''
#* Go to '''Connectivity''' &gt; '''Outbound Routes'''.
#* Click '''+ Add Outbound Route'''.
# '''Set Up the Outbound Route'''
#* '''Route Name''': Enter <code>UniTel_Outbound</code>.
#* '''Route CID''': Enter your UniTel DID (in e.164 format, e.g., 13475522258).
# '''Assign Trunk to Route'''
#* '''Trunk Sequence for Matched Routes''': Select <code>UniTel_SIP</code> (the trunk created earlier). We don’t have multiple trunks.
# '''Navigate to Dial Patterns'''
#* Click '''Dial patterns wizards'''.
#* Click the dial plans that make sense for your locale.
#* This is a conversation for you to have with your SIP trunking provider based on your region. Open a ticket with them and make sure you choose the right options here!
# '''Submit and Apply'''

* Click '''Submit'''.
* Click '''Apply Config''' to save and activate the outbound route.

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:lu67917r1ezu_tmp_ef8d201c.png
File:lu67917r1ezu_tmp_58e2fa70.png
File:lu67917r1ezu_tmp_4bc11745.png
File:lu67917r1ezu_tmp_1252fdc2.png
File:lu67917r1ezu_tmp_2106a642.png
File:lu67917r1ezu_tmp_7330e060.png
</gallery>

<span id="step-7-configure-ip-subnets-in-freepbx"></span>
== Step 7: Configure IP subnets in FreePBX ==

FreePBX configures it to work automatically with the LAN subnet. For instance, if you chose <code>192.168.5.0/24</code> for your local network, it will configure FreePBX to work properly with your LAN subnet.

However, it doesn’t know you have a VPN. Remember that I suggest you not open ports. If you want this to work on your Android phone or iPhone when you connect to your home network, you have to add your VPN subnet manually. To do so, follow these instructions:

<span id="add-vpn-subnet-to-local-networks-in-freepbx"></span>
==== Add VPN subnet to local networks in FreePBX ====

# '''Navigate to SIP Settings'''
# Go to '''“Settings”''' &gt; '''“Advanced SIP Settings”'''.
# Make sure you are on the '''“General”''' tab.
# Find the '''Local Networks''' section.
# Log into the '''pfSense''' firewall in a new browser tab.
# Go to '''“VPN” —&gt; “OpenVPN”''' at the top menu.
# Find the '''Tunnel Network''' for your VPN, which will be in the list of '''OpenVPN servers'''.
# Return to the FreePBX browser tab and click '''Add Local Network Field'''.
# Add the '''Tunnel Network''' of your VPN.
# '''Submit and Apply Changes'''
#* Click '''“Submit”'''.
#* Click '''“Apply Config”''' to activate the outbound route.

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:lu67917r1ezu_tmp_ce66bf80.png
File:lu67917r1ezu_tmp_2e61fc14.png
File:lu67917r1ezu_tmp_b500d2da.png
</gallery>

<span id="step-8-setting-up-a-softphone"></span>
== Step 8: Setting up a softphone ==

A softphone is a software phone. I’ll show you how to use this; the instructions are about the same as configuring a hardware phone. It also allows me to create instructions that allow anyone watching this to confirm their system works without having to provide generalized instructions that aren’t precise to every single smartphone.

<span id="download-zoiper"></span>
==== 8.1 Download Zoiper ====

# Go to Zoiper.
# Download Zoiper [https://www.zoiper.com/ here.]
# Install Zoiper.
# Run Zoiper.

<span id="get-credentials-for-your-extension."></span>
==== 8.2 Get credentials for your extension. ====

* '''Log in to the FreePBX Admin Interface'''
* '''Navigate to Extensions'''
** Go to '''Connectivity &gt; Extensions'''.
** Click your extension.
* Get your '''extension number''', which is your '''username''', and your '''secret''', which is your '''password'''.

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:lu67917r1ezu_tmp_5a55c888.png
File:lu67917r1ezu_tmp_4285096.png
File:lu67917r1ezu_tmp_3955f9e6.png
File:lu67917r1ezu_tmp_c8bb85.png
File:lu67917r1ezu_tmp_f287090.png
</gallery> 8.3 Configure Zoiper

# Open Zoiper and select '''Create New Account'''.
# Enter the following details:
#* '''Username''': Your extension number (e.g., 401).
#* '''Password''': Your secret (password).
#* '''Domain''': Your server’s IP address or hostname (for us, <code>192.168.5.6:5060</code>).
#** The format will be <code>username@ip:port</code> with the '''password''' in its own field.
#** In my setup with my example, it would be [http://401@192.168.5.6 '''401@192.168.5.6''']:5060 or [http://401@freepbx.home.arpa '''401@freepbx.home.arpa''']:5060.
#* Skip Outbound Proxy.
#* Select Transport Protocol '''SIP UDP'''.
#* Choose SIP UDP as most setups use UDP by default.
# Test Configuration:
#* If you see a green checkmark, you did good.

<span id="test-audio-settings"></span>
==== 8.4 Test Audio Settings ====

# Go to options.
# Select Input and Output Devices.
# '''Set Input Device''' to your microphone.
# '''Set Output Device''' to your speakers or headphones.
# '''Test Audio''': Speak into the microphone to check input levels. For output, press play to confirm audio works.

<span id="step-9-configuring-voicemail-and-email-notifications-in-freepbx-17"></span>
== Step 9: Configuring Voicemail and Email Notifications in FreePBX 17 ==

This guide provides step-by-step instructions to configure voicemail for an extension and ring group in FreePBX 17, making sure that voicemail messages are sent via email with audio file attachments. We will also set up your custom mail server for sending these email notification; we’re not calling into a voicemail system in 2024.

<span id="enable-voicemail-for-the-extension"></span>
==== 9.1 Enable Voicemail for the Extension ====

# '''Log in to the FreePBX Admin Interface'''
#* <code>http://192.168.5.5/admin</code>
#* Log in with your admin credentials.
# '''Navigate to Extensions'''
#* Go to '''“Applications”''' &gt; '''“Extensions”'''.
#* Find and select the extension (e.g., <code>401</code>) that you want to set up.
# '''Enable Voicemail for the Extension'''
#* Scroll down to the '''Voicemail''' section.
#* '''Enable Voicemail''': Set to <code>Yes</code>.
#* '''Voicemail Password''': Enter a numerical for accessing voicemail which we will never use.
#* '''Email Address''': Enter the email address where voicemail notifications should be sent (e.g., <code>user@example.com</code>).
#* '''Attach Voicemail''': Set to <code>Yes</code> (this will attach the audio file of the voicemail to the email notification).
#* '''Delete Voicemail''': Set to <code>No</code> (keeps a copy of the voicemail on the system even after sending the email). Until we know if our system works, keep this to no. Once it is emailing us our voicemails as a wav file, then we can change this to yes.
# '''Submit and Apply Changes'''
#* Click '''“Submit”'''.
#* Click '''“Apply Config”''' to save the voicemail settings for the extension.

<span id="configure-voicemail-for-the-ring-group"></span>
==== 9.2 Configure Voicemail for the Ring Group ====

# '''Navigate to Ring Groups'''
#* Go to '''“Applications”''' &gt; '''“Ring Groups”'''.
#* Select the ring group you configured earlier (e.g., <code>600 - Office Ring Group</code>).
# '''Set Ring Group to Go to Voicemail'''
#* '''Destination if No Answer''': Choose '''“Voicemail”''', and select the extension’s voicemail (e.g., <code>101</code>).
# '''Submit and Apply Changes'''
#* Click '''“Submit”'''.

<span id="step-9.3-configure-freepbx-to-send-email-notifications-via-custom-mail-server"></span>
==== Step 9.3: Configure FreePBX to Send Email Notifications via Custom Mail Server ====

# '''Navigate to System Admin Module'''
#* Go to '''“Admin”''' &gt; '''“System Admin”'''.
#* Click on '''“Email Setup”'''…. '''''GOTCHA!!''''' This is GNU/Linux, nothing is easy. I had you for a moment there, didn’t I? :D
# This is actually going to be a fun journey of configuring postfix manually. That is a long way away, at the end.

<span id="step-10-setting-up-pfsense-firewall-rules-for-freepbx-with-unitel-sip-services"></span>
== Step 10: Setting Up pfSense Firewall Rules for FreePBX with UniTel SIP Services ==

To make sure your FreePBX system (located at <code>192.168.5.6</code>) is able to connect to UniTel’s SIP service and receive calls with two-way audio that actually work, we need to create NAT rules &amp; corresponding firewall rules that only allow traffic from UniTel’s approved IPs. Next we’ll walk you through setting up aliases for UniTel’s IPs, creating NAT rules, &amp; making sure SIP and RTP traffic flows correctly.

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:lu67917r1ezu_tmp_694465ad.png
File:lu67917r1ezu_tmp_f3d60c6e.png
</gallery> Create aliases for UniTel’s IPs

<span id="step-10.1-log-in-to-your-pfsense-web-interface"></span>
==== Step 10.1: Log in to Your pfSense Web Interface ====

# Open a web browser and navigate to: <code>https://pfsense.home.arpa</code> or [https://192.168.5.1/ https://192.168.5.1]
# Enter your pfSense admin credentials.

<span id="step-10.2-add-an-alias-for-unitels-sip-signaling-ips"></span>
==== Step 10.2: Add an Alias for UniTel’s SIP Signaling IPs ====

# Go to '''Firewall''' &gt; '''Aliases'''.
# Click '''Add''' (+) to create a new alias.
# Configure the alias as follows:
#* '''Name''': <code>Unitel_SIP_IPs</code>
#* '''Description''': <code>SIP Signaling IPs from UniTel</code>
#* '''Type''': <code>Host(s)</code>
#* '''IP Addresses''':
Add each of the following SIP IP addresses: '''''THIS MAY CHANGE, CHECK UNITEL GETTING STARTED PAGE TO MAKE SURE THESE ARE THE RIGHT ONES!'''''
#** <code>199.180.220.89</code>
#** <code>199.180.220.91</code>
#** <code>208.89.104.3</code>
# Click '''Save''', then '''Apply Changes'''.

<span id="step-10.3-add-an-alias-for-unitels-media-ips"></span>
==== Step 10.3: Add an Alias for UniTel’s Media IPs ====

# In the '''Aliases''' section, click '''Add''' again to create another alias.
# Configure the alias as follows:
# '''Name''': Unitel_Media_IPs
# '''Description''': Media IPs for UniTel SIP Services
# '''Type''': Host(s)
# '''IP Addresses''': Add each of the following media IP addresses: '''''THIS MAY CHANGE, CHECK UNITEL GETTING STARTED PAGE TO MAKE SURE THESE ARE THE RIGHT ONES!'''''
#* 199.180.223.109
#* 45.55.33.77
#* 157.230.238.197
#* 45.33.70.196
#* 45.33.71.83
#* 159.65.107.252
#* 45.33.14.21
#* 159.89.122.218
#* 167.71.237.189
#* 172.104.226.108
#* 139.162.250.71
# Click '''Save''', then '''Apply Changes'''.

<span id="setting-up-nat-port-forward-firewall-rules"></span>
=== Setting up NAT port forward &amp; firewall rules ===

<div class="figure">

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:lu67917r1ezu_tmp_3c485302.png
</gallery>

</div>
<div class="figure">

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:image-20241107012035037.png
</gallery>

</div>
<div class="figure">

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:image-20241107012048982.png
</gallery>

</div>
<div class="figure">

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:image-20241107012056979.png
</gallery>

</div>
<span id="configure-nat-port-forwards-for-freepbx-signalling"></span>
==== 10.4 Configure NAT port forwards for FreePBX signalling ====

# Navigate to '''Firewall''' &gt; '''NAT'''.
# Under the '''Port Forward''' tab, click '''Add''' to create a new NAT rule.
# Configure the rule as follows:
#* '''Interface''': WAN
#* '''Protocol''': UDP
#* '''Destination''': WAN address
#* '''Destination Port Range''':
#** From: 5060
#** To: 5065 (for SIP signaling)
#* '''Redirect Target IP''': Enter your PBX IP: 192.168.5.6
#* '''Redirect Target Port''':
#** From: 5060
#** To: 5065
#* '''Source''': Select Single host or alias and choose Unitel_SIP_IPs.
#* '''Description''': Forward SIP Traffic from UniTel to FreePBX
# Click '''Save''', then '''Apply Changes'''.

<div class="figure">

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:image-20241107012336127.png
</gallery>

</div>
<span id="set-up-nat-port-forwards-for-rtp-media-traffic"></span>
==== 10.5 Set Up NAT port forwards for RTP (Media) Traffic ====

# In the '''Port Forward''' tab, click '''Add''' to create another NAT rule.
# Configure the rule as follows:
#* '''Interface''': WAN
#* '''Protocol''': UDP
#* '''Destination''': WAN address
#* '''Destination Port Range''':
#** From: 10000
#** To: 20000 (for RTP media traffic)
# '''Redirect Target IP''': Enter your PBX IP: 192.168.5.6
# '''Redirect Target Port''':
#* From: 10000
#* To: 20000
# '''Source''': Select Single host or alias and choose <code>Unitel_Media_IPs</code>.
# '''Description''': Forward RTP Traffic from UniTel to FreePBX
#* Click '''Save''', then '''Apply Changes'''.

<span id="verify-automatic-firewall-rules"></span>
==== 10.6 Verify Automatic Firewall Rules ====

# After creating the NAT rules, go to '''Firewall''' &gt; '''Rules'''.
# In the '''WAN''' tab, confirm that the firewall rules were automatically created for:
## '''SIP Traffic''' (ports 5060-5065) pointing to 192.168.5.6 and restricted to <code>Unitel_SIP_IPs</code>.
## '''RTP Traffic''' (ports 10000-20000) pointing to 192.168.5.6 and restricted to <code>Unitel_Media_IPs</code>.

<span id="test-the-configuration"></span>
==== 10.7 Test the Configuration ====

# Make sure that your FreePBX system can register with UniTel’s SIP servers.
# Make a test call to make sure both SIP signaling and media (audio) traffic are functioning correctly.
# Make sure that when you end a phone call, both the caller &amp; recipient notice that it has ended immediately.
# Make sure you have two-way audio.
# Leave a call on for fifteen minutes and make sure it doesn’t hang up by itself.

<span id="step-11-troubleshooting-when-it-doesnt-work.-its-open-source-so."></span>
== Step 11: Troubleshooting when it doesn’t work. It’s open source, so…. ==

<span id="introduction-to-network-rules"></span>
=== Introduction to Network Rules ===

We set up two sets of rules:

# '''SIP Trunk rules''' (Ports 5060-5065)
#* Allows Unitel to talk to our PBX
#* Deals with signaling &amp; connection management
# '''Media Proxy Rules''' (Ports 10000-20000)
#* Manages the actual audio transmission
#* Handles voice data going back and forth

<span id="what-are-nat-port-forwards-vs-firewall-rules"></span>
==== What are NAT port forwards vs Firewall Rules? ====

<span id="network-address-translation-nat-port-forwards"></span>
===== Network Address Translation (NAT) Port Forwards =====

NAT is like the restaurant host who brings guests to specific tables. It allows specific machines behind your network to get traffic depending on the port that the traffic was trying to access when the traffic got to your cable modem &amp; firewall.

<span id="firewall-rules-1"></span>
===== Firewall Rules =====

The firewall acts as a bouncer. Even when NAT directs traffic to the right computer, the firewall can still block problematic connections.

<span id="order"></span>
===== Order: =====

<code>pfSense</code> will add a firewall rule AUTOMATICALLY each time you create a NAT port forward, as long as you do not change that option at the end of the NAT port forward rule creation page. I circled this to make sure you would get it right.

# Set up NAT rules first
# Configure firewall rules second

<span id="our-setup"></span>
===== Our Setup =====

FreePBX box IP address: 192.168.5.6

Internet Traffic → NAT (Traffic Direction) → Firewall (Security Check) → FreePBX virtual machine

<span id="when-things-dont-work-common-scenario"></span>
=== When Things Don’t Work (Common Scenario) ===

This is an open source firewall combined with self-managed SIP trunking. If something works on the first go, you should be very concerned – this likely means you are in a coma &amp; dreaming. Try to wake up. If you can’t, something is wrong.

<blockquote>'''IMPORTANT:''' Follow along in the video as this is best explained there as I go. This is one of the few sections where I believe the video is a must-have to understand how troubleshooting an issue here would work in real time.
</blockquote>
When initial setup doesn’t work, follow this troubleshooting sequence:

# '''Clear ARP Tables'''
#* Navigate to '''Diagnostics → ARP Table → Clear'''
# '''Reset States'''
#* Navigate to '''Diagnostics → States → Reset States'''
#* States are current connections
#* Must be reset on both routers
#* Wait 90 seconds after reset (best practice)
# '''Reload Filter Rules'''
#* Navigate to '''Status → Filter →''' then click ** Reload**

<span id="using-packet-capture-for-diagnostics"></span>
==== Using Packet Capture for Diagnostics ====

# Go to: '''Diagnostics → Packet Capture'''
# Configure capture:
#* Interface: WAN or LAN depending on test
#* Port: 5060, 5061, 5062, 5063, 5064, 5065 for SIP traffic

<span id="reading-packet-capture-results"></span>
==== Reading Packet Capture Results ====

* Example of captured traffic: 199.18.220.89 (Unitel’s IP in my case)
* You’re looking to see if the port 5060 traffic is actually being directed to your PBX. You’re also looking to see if it is even coming in at all.

<span id="stuff-we-use-to-troubleshoot"></span>
=== Stuff we use to troubleshoot: ===

When dealing with miserable issues:

<ol style="list-style-type: decimal;">
<li><p>'''Check Logs'''</p>
<p>'''Status → System Logs → Firewall → Normal View'''</p>
<ul>
<li>Sort by newest first
</li>
<li>Enable logging for allowed and blocked traffic</li></ul>
</li>
<li><p>'''Use diagnosing tools'''</p>
<ul>
<li><p>Packet capture shows where things are going</p></li>
<li><p>Firewall logs show what’s being blocked/allowed</p></li>
<li><p>Side-by-side comparison of rules vs. actual traffic</p></li></ul>
</li>
<li><p>'''Reset Everything'''</p>
<ul>
<li><p>Clear ARP tables</p></li>
<li><p>Reset state tables</p></li>
<li><p>Reload filter rules</p></li>
<li><p>None of this will work because it’s open source, SO:</p></li>
<li><p>Reboot the router</p></li>
<li><p>Look for hints &amp; clues.</p></li></ul>
</li></ol>

<span id="important-takeaway-from-this"></span>
=== Important takeaway from this ===

* In the video, I did all of the above. The router magically started passing traffic after a reboot.
* Even when everything is configured correctly, it may not work correctly – it’s open source.
* Consumer routers vs Enterprise/Open Source firewalls:
** $20 consumer router: “It just works”
** Enterprise-grade open source firewall: Requires patience and systematic troubleshooting
* It’s still better to use this than a traditional router so you don’t get hacked &amp; owned via lack of updates.

<span id="step-12-install-lenny-on-freepbx-17"></span>
== Step 12: Install Lenny on FreePBX 17 ==

<span id="prepare-to-access-your-freepbx-system"></span>
==== 12.1 Prepare to Access Your FreePBX System ====

You need to SSH into the FreePBX VM to install Lenny. Open a terminal on your local machine and connect via SSH:

<pre>ssh louis@192.168.5.6
su</pre>
<span id="add-lennys-custom-context-in-asterisk"></span>
==== 12.2 Add Lenny’s Custom Context in Asterisk ====

<ol style="list-style-type: decimal;">
<li><p>Open the <code>extensions_custom.conf</code> file for editing:</p>
<pre>nano /etc/asterisk/extensions_custom.conf</pre></li>
<li><p>Add the following lines to define the Lenny context:</p></li></ol>

<pre>[Lenny]
exten =&gt; talk,1,Set(i=${IF($[&quot;0${i}&quot;=&quot;016&quot;]?7:$[0${i}+1])})
same =&gt; n,ExecIf($[${i}=1]?MixMonitor(${UNIQUEID}.wav))
same =&gt; n,Playback(Lenny/Lenny${i})
same =&gt; n,BackgroundDetect(Lenny/backgroundnoise,1500)</pre>
<ol start="3" style="list-style-type: decimal;">
<li>Save and exit the editor by pressing <code>Ctrl + X</code>, then <code>Y</code>, and <code>Enter</code>.</li></ol>

<span id="download-lennys-sound-files"></span>
==== 12.3 Download Lenny’s Sound Files ====

Continue with the necessary steps to download and configure Lenny’s sound files as required.

Lenny works by playing recorded audio. You’ll need to download these audio files to the correct directory on your FreePBX system.

Download Lenny’s sound files [https://www.crosstalksolutions.com/pub/Lenny.zip from this link]. Crosstalk solutions is a hero for continuing to host this. Tell him thank you.

<pre>cd /var/lib/asterisk/sounds/
wget https://www.crosstalksolutions.com/pub/Lenny.zip
unzip Lenny.zip
chown asterisk:asterisk /var/lib/asterisk/sounds/Lenny/* -R
chmod -R 755 /var/lib/asterisk/sounds/Lenny/* -R</pre>
<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:lu67917r1ezu_tmp_89b069de.png
File:lu67917r1ezu_tmp_6b11bfcb.png
File:lu67917r1ezu_tmp_410d95bf.png
File:lu67917r1ezu_tmp_cdc425ee.png
</gallery>

<span id="create-a-custom-destination-in-freepbx"></span>
==== 12.4 Create a Custom Destination in FreePBX ====

# Log in to the FreePBX web interface.
# Navigate to '''Admin &gt; Custom Destinations'''.
# Add a new custom destination with the following details:
#* '''Custom Destination:''' <code>Lenny,talk,1</code>
# This may be called TARGET instead of CUSTOM DESTINATION IN NEW VERSIONS
#* '''Description:''' Lenny
# Click '''Submit''' and then '''Apply Config''' to save the changes.

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:lu67917r1ezu_tmp_5e139a78.png
File:lu67917r1ezu_tmp_35e121d2.png
File:lu67917r1ezu_tmp_91b7118c.png
File:lu67917r1ezu_tmp_77d50bb9.png
File:lu67917r1ezu_tmp_c6db1985.png
</gallery>

<span id="set-lenny-as-a-destination"></span>
==== 12.5 Set Lenny as a Destination ====

You now have multiple options for how to use Lenny.

* '''Manual Transfers to Lenny:'''
** Navigate to '''Connectivity —&gt; Extensions'''
** Create a new '''Virtual Extension'''
** Set the extension to whatever you want it to be; this is the number you dial to get Lenny &amp; the number you transfer people to for Lenny
** Click on the Advanced tab
** Scroll to the bottom for destinations when nobody answers.
** Set each of the three to '''Custom Destinations —&gt; Lenny'''
** Enjoy transferring telemarketers to Lenny at his extension. :)

<span id="reload-things-in-the-terminal."></span>
==== 12.6 Reload things in the terminal. ====

In your SSH terminal, type the following:

<pre>fwconsole reload</pre>
<blockquote>'''IMPORTANT:''' Hitting the red '''“Apply Config”''' button in the upper right corner of the FreePBX webpage is not enough here. For this to work, you must run <code>fwconsole reload</code> in the terminal.
</blockquote>
<span id="sending-blocked-numbers-to-lenny"></span>
==== 12.7 Sending blocked numbers to Lenny ====

# After hanging up on someone you hate, hit <code>*32</code> quickly which will block their number.
# Navigate to '''Admin —&gt; Blacklist'''.
# Click onto '''Settings'''.
# Set the '''Destination for BlackListed Calls''' to '''Custom Destination —&gt; Lenny'''.

Now every time you get a call from someone you hate, you can dial <code>*32</code> &amp; they will be routed to Lenny as soon as they call back. But remember, in the words of one of my first recording studio job bosses in 2007 - ''“Louis, you hate nothing; you intensely dislike it!”''

<span id="step-12-hiring-a-virtual-receptionist-who-tells-annoying-people-to-get-the-fuck-outta-here"></span>
== Step 12: Hiring a virtual receptionist who tells annoying people to ''“get the fuck outta here!”'' ==

This is the primary reason to have a self managed PBX.

<span id="download-the-sound-files"></span>
==== 12.1 Download the Sound Files ====

First, SSH into your FreePBX machine:

<pre>ssh louis@freepbx.home.arpa</pre>
Download the sound files from the given URL using <code>wget</code>:

<pre>wget http://downloads.asterisk.org/pub/telephony/sounds/asterisk-extra-sounds-en-g722-current.tar.gz</pre>
<span id="place-the-files-in-the-proper-directory"></span>
==== 12.2 Place the Files in the Proper Directory ====

After downloading the archive, extract it and move the files to the appropriate directory in FreePBX. Asterisk sound files typically reside in <code>/var/lib/asterisk/sounds</code>.

Extract the tarball:

<pre>tar -xvzf asterisk-extra-sounds-en-g722-current.tar.gz</pre>
Move the extracted files into the custom sound directory for FreePBX:

<pre>mv asterisk-extra-sounds-en-g722 /var/lib/asterisk/sounds/custom</pre>
If the <code>custom</code> directory doesn’t exist, you can create it:

<pre>mkdir /var/lib/asterisk/sounds/custom</pre>
<span id="set-correct-permissions"></span>
==== 12.3 Set Correct Permissions ====

Make sure that FreePBX and Asterisk can access the sound files by setting the correct ownership and permissions. FreePBX generally runs under the <code>asterisk</code> user:

<pre>chown -R asterisk:asterisk /var/lib/asterisk/sounds/custom/* -R
chmod -R 755 /var/lib/asterisk/sounds/custom/* -R</pre>
<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:lu67917r1ezu_tmp_3725236e.png
File:lu67917r1ezu_tmp_8a2cf792.png
File:lu67917r1ezu_tmp_2abf75f3.png
</gallery> 12.4 Find the Sound Files in the FreePBX GUI

# Log in to the '''FreePBX Admin Interface'''.
# Navigate to '''Admin &gt; System Recordings'''.
# Under '''Add Recording''', you should now be able to see &amp; use the uploaded sound files from the <code>/var/lib/asterisk/sounds/custom</code> directory.

<span id="combine-sound-prompts-into-a-sequence"></span>
==== 12.5 Combine Sound Prompts into a Sequence ====

To combine multiple sound files into a single prompt sequence, use the '''System Recordings''' feature in FreePBX:

# Go to '''Admin &gt; System Recordings''' and create a new recording.
#* Select the option to '''Add Sound Recording''' by combining the existing files.
#* Add the sound files in the order you want them to play.
# Choose the following codecs:
#* alaw
#* g722
#* gsm
#* ulaw
#* wav
# EXCLUDE the following codecs:
#* g729
#* sln
#* sln16
#* sln48
# Save the combined sound as a new recording.

<span id="create-an-extension-that-plays-the-sound-prompts"></span>
==== 12.6 Create an Extension That Plays the Sound Prompts ====

To forward someone to an extension that plays back the sound prompts:

# Log in to the '''FreePBX Admin Interface'''.
# Navigate to '''Applications &gt; Extensions'''.
# Click '''Add Extension''' and select '''Custom Extension'''.
# Set destination of unanswered to play your recording.
# '''Save, Submit''' and '''Apply Config'''.

Now, you can transfer calls to this extension, and the selected sound prompts will be played back. Allison Smith will tell.

<span id="step-13-get-emails-with-voicemails-using-postfix-with-postmark-smtp-relay"></span>
== Step 13: Get emails with voicemails using Postfix with Postmark SMTP Relay ==

We are not doing the 1990s calling into voicemail system nonsense. That is miserable.

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:lu67917r1ezu_tmp_aaeb276e.png
File:lu67917r1ezu_tmp_8c59ac9.png
File:lu67917r1ezu_tmp_8cb7f78.png
</gallery>

<span id="configure-the-from-address-in-freepbx"></span>
==== 13.1 Configure the FROM Address in FreePBX ====

# Log into your FreePBX web interface.
# Navigate to '''Settings → Voicemail Admin'''.
# Click the '''Settings''' tab.
# Click on the '''Email Config''' tab.
# Set the '''Server Email''' to an email address of your choice.
#* I suggest this address be within the domain of the email you set up in mailcow.
#* For instance, if you set up an email for yourself called [http://steve@stevesavers.com '''steve@stevesavers.com'''] in mailcow, make this '''voicemails@stevesavers.com'''.
# Click '''Submit''', then '''Apply Config''' (red button in the upper right corner).

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:lu67917r1ezu_tmp_7c7fe30c.png
File:lu67917r1ezu_tmp_926f2cd4.png
File:lu67917r1ezu_tmp_82ab253e.png
File:lu67917r1ezu_tmp_302269b1.png
File:lu67917r1ezu_tmp_5b69f714.png
File:lu67917r1ezu_tmp_4ea2455d.png
</gallery>

<span id="configure-user-access-to-voicemail"></span>
==== 13.2 Configure user access to voicemail ====

# Navigate to '''Admin —&gt; User Management'''.
# Click '''Edit''' next to the user.
#* Click the '''User Details''' tab at the top.
#** Check that the email address is correct.
#* Click the '''UCP''' tab at the top.
#** Click the '''Call History''' sub-tab.
#*** In '''CDR Access''', add the extensions for which you want to allow this user to listen to call recordings. So if your extension is '''401''', then '''401''' should be in this list.
#*** Set '''Allow CDR''' to '''Yes'''.
#*** Set '''Allow CDR Downloads''' to '''Yes'''.
#*** Set '''Allow CDR Playback''' to '''Yes'''.
#** Click the '''Voicemail''' sub-tab.
#*** Make sure every option here is set to '''Yes'''.
#*** In '''Allowed Voicemail''', make sure that your extension is in the list. So if your extension is '''401''', then '''401''' should be in this list.
#* Click '''Submit''', then '''Apply Config''' (red button in the upper right corner).

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:lu67917r1ezu_tmp_dd6f816d.png
File:lu67917r1ezu_tmp_653952c.png
File:lu67917r1ezu_tmp_d77e3b32.png
File:lu67917r1ezu_tmp_56847681.png
File:lu67917r1ezu_tmp_aa85fde6.png
File:lu67917r1ezu_tmp_6bcaece6.png
File:lu67917r1ezu_tmp_abb6e1ff.png
File:lu67917r1ezu_tmp_df1ec9c8.png
</gallery>

<span id="configure-extension-for-voicemail"></span>
==== 13.3 Configure extension for voicemail ====

# Navigate to '''Connectivity —&gt; Extensions'''
#* Choose your extension
# Go to '''Voicemail'''
#* Set your '''Voicemail password'''
#* Set the '''Email Address''' to the email address you want it to email.
#* Click '''Submit''', then '''Apply Config''' red button in the upper right corner
# Click '''UCP''' on the top menu to enter the User Control Panel
#* Click the plus sign in the upper left to add a panel.
#* Choose '''Voicemail.'''
#* Choose your extension, in this case, '''401'''
#* Go to the little gear on the upper right corner of the panel you just added to open the '''settings''' menu
#* Make sure '''Email Attachment''' is '''On'''
#* '''Email Address''' should be the address that you want voicemails to go to.

<span id="get-postmark-credentials-for-smtp-relay"></span>
==== 13.4 Get Postmark Credentials for SMTP relay ====

We are using Postmark for SMTP relay so our emails are not immediately rejected by most providers.

# Go to postmarkapp.com
# Log in and click Servers
# Click onto the server you made earlier.
# Click '''Default Transactional Stream'''
# Navigate to the '''Setup Instructions page''' after clicking onto your message stream.
#* Under '''“Pick the library or integration”''' – pick “SMTP”.
#* '''This is the same thing we did when we set up mailcow with Postmark for SMTP relay in the mailcow section!'''
#* Take note of these, as we will be using them with FreePBX

<span id="modify-postfix-configuration"></span>
==== 13.5 Modify Postfix Configuration ====

<ol style="list-style-type: decimal;">
<li><p>Edit the main configuration file:</p>
<pre>sudo nano /etc/postfix/main.cf</pre></li>
<li><p>Find and modify/add these lines. Keep everything elsein the <code>main.cf</code> file unchanged. Adjust the <code>sender_canonical_maps = static:voicemailalert@stevesavers.com</code> to the email address you wish to use.</p></li></ol>

<pre>relayhost = [smtp.postmarkapp.com]:587
smtp_use_tls = yes
smtp_sasl_auth_enable = yes
smtp_sasl_password_maps = hash:/etc/postfix/sasl_passwd
smtp_sasl_security_options = noanonymous
smtp_sasl_mechanism_filter = plain
sender_canonical_maps = static:voicemailalert@stevesavers.com</pre>
<span id="set-up-authentication"></span>
==== 13.6 Set Up Authentication ====

<ol style="list-style-type: decimal;">
<li><p>Ssh into the FreePBX virtual machine:</p>
<pre>ssh louis@192.168.5.6</pre></li>
<li><p>Create/edit the SASL password file:</p>
<pre>sudo nano /etc/postfix/sasl_passwd</pre></li>
<li><p>Add this line (replace <code>USERNAME:PASSWORD</code> with your Postmark credentials):</p>
<pre>[smtp.postmarkapp.com]:587 USERNAME:PASSWORD</pre></li>
<li><p>Create the hash database and set permissions:</p>
<pre>sudo postmap /etc/postfix/sasl_passwd
sudo chmod 600 /etc/postfix/sasl_passwd*</pre></li></ol>

<span id="restart-postfix"></span>
==== 13.7 Restart Postfix ====

<pre>sudo systemctl restart postfix</pre>
<span id="test-configuration"></span>
==== 13.8 Test Configuration ====

Send a test email:

<pre>cat &lt;&lt; EOF | sendmail l.a.rossmann@gmail.com
From: voicemailalert@stevesavers.com
To: l.a.rossmann@gmail.com
Subject: Test Email
Content-Type: text/plain
X-PM-Message-Stream: outbound

This is a test email body.
EOF</pre>
Check mail logs for errors:

<pre>sudo tail -f /var/log/mail.log</pre>
<span id="troubleshooting"></span>
==== Troubleshooting ====

If emails aren’t sending:

# Check <code>/var/log/mail.log</code> for errors
# Check that Postmark credentials are correct (if you typed <code>postmark.com</code> instead of <code>postmarkapp.com</code> for server, etc)
# Verify sender domain (<code>stevesavers.com</code>) is properly configured in Postmark
# Check the activity tab on the transactional stream in Postmark
# Mail log will tell you what you fkd up 99% of time.

<div class="figure">

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:lu67917r1ezu_tmp_923a1598.png
</gallery>

</div>
!(Postmark Activity monitor](old/images/lu67917r1ezu_tmp_f60bd933.png)

!(Postmark Activity monitor](old/images/lu67917r1ezu_tmp_c39a116d.png)

<span id="postmark-activity-monitor"></span>
===== Postmark Activity Monitor: =====

If you want more troubleshooting information, check Postmark.

# Log into Postmark.
# Click '''Servers'''
# Click onto the server you made.
# Click onto your '''Default Transactional Stream'''
# Click '''Activity'''
# Poke around.

<span id="default-etcpostfixmain.cf-config-file"></span>
==== Default /etc/postfix/main.cf config file ====

Just in case you mess something up, here’s the default one, because the ones in /usr/share/postfix require configuration from scratch. What they mean when they say ''“more complete”'' version is ''“we don’t offer a copy anywhere of the just working version”'', because it’s… GNU/Linux.

<pre># See /usr/share/postfix/main.cf.dist for a commented, more complete version


# Debian specific:  Specifying a file name will cause the first
# line of that file to be used as the name.  The Debian default
# is /etc/mailname.
#myorigin = /etc/mailname

smtpd_banner = $myhostname ESMTP $mail_name (Debian/GNU)
biff = no

# appending .domain is the MUA's job.
append_dot_mydomain = no

# Uncomment the next line to generate &quot;delayed mail&quot; warnings
#delay_warning_time = 4h

readme_directory = no

# See http://www.postfix.org/COMPATIBILITY_README.html -- default to 3.6 on
# fresh installs.
compatibility_level = 3.6



# TLS parameters
smtpd_tls_cert_file=/etc/ssl/certs/ssl-cert-snakeoil.pem
smtpd_tls_key_file=/etc/ssl/private/ssl-cert-snakeoil.key
smtpd_tls_security_level=may

smtp_tls_CApath=/etc/ssl/certs
smtp_tls_security_level=may
smtp_tls_session_cache_database = btree:${data_directory}/smtp_scache


smtpd_relay_restrictions = permit_mynetworks permit_sasl_authenticated defer_unauth_destination
myhostname = debian.home.arpa
alias_maps = hash:/etc/aliases
alias_database = hash:/etc/aliases
mydestination = $myhostname, debian, localhost.localdomain, localhost
relayhost = 
mynetworks = 127.0.0.0/8 [::ffff:127.0.0.0]/104 [::1]/128
mailbox_size_limit = 0
recipient_delimiter = +
# WARNING: Changing the inet_interfaces to an IP other than 127.0.0.1 may expose Postfix to external network connections.
# Only modify this setting if you understand the implications and have specific network requirements.
inet_interfaces = 127.0.0.1
inet_protocols = all
message_size_limit = 102400000</pre>
<span id="self-hosted-bitwarden-password-manager"></span>
= '''Self-Hosted Bitwarden Password manager:''' =

<span id="this-is-a-bad-idea."></span>
== This is a bad idea. ==

We are going to set this up on our mailcow virtual machine at 192.168.5.3.

This is a bad idea. You shouldn’t do this. Not only are you starting off as a beginner self-managing something that literally is the key to every aspect of your life, but you aren’t even saving money. Simple basics like the <code>haveibeenpwned</code> integration to check for leaked passwords will cost you more to do yourself when self-hosting than it would if you just paid Bitwarden.

A big reason we’re doing this is freedom; we want freedom from crappy companies. Bitwarden isn’t a bad company. They treat users well, and they give you the freedom to self-host your own instance with software they’ve open-sourced. If anything, these are the types of companies that have done more to earn the public’s trust than the rest.

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:lu67917r1ezu_tmp_8197e237.png
File:lu67917r1ezu_tmp_8ad58b71.png
File:lu67917r1ezu_tmp_251d8ba7.png
</gallery>

<span id="step-1-configure-dns-resolution-in-pfsense"></span>
== Step 1: '''Configure DNS Resolution in pfSense''' ==

Before installing Bitwarden, we should configure DNS resolution since our server (192.168.5.3) already resolves to <code>mailserver.home.arpa</code>.

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:lu67917r1ezu_tmp_6b44e385.png
File:lu67917r1ezu_tmp_b1a9776a.png
File:lu67917r1ezu_tmp_32ced0e0.png
</gallery> '''Add Additional DNS Entry'''

# Log into your pfSense dashboard.
# Navigate to '''Services''' &gt; '''DNS Resolver'''.
# Scroll down to '''Host Overrides'''.
# Click the plus ('''+''') button to add a new entry.
# Fill in the following:
#* '''Host''': <code>bitwarden</code>
#* '''Domain''': <code>home.arpa</code>
#* '''IP Address''': <code>192.168.5.3</code>
#* '''Description''': Bitwarden Password Manager
# For '''Additional Names for this Host''':
#* '''Host name''' should be <code>mailserver</code> since 192.168.5.3 is also our mailserver and already has a static mapping as a mailserver.
#* '''Domain''' should be <code>home.arpa</code> (or whatever you set as your domain in '''System —&gt; General Settings''').
#* '''Description''' can be anything you want.
# Click '''Save'''.
# Click '''Apply Changes'''.

'''Note''': This server will now respond to both <code>mailserver.home.arpa</code> and <code>bitwarden.home.arpa</code>.

'''Step 2 below is only necessary if you did NOT follow these while you were setting up this virtual machine for <code>mailcow</code> mailserver. Skip ahead to Step 3 if you already did this when setting up <code>mailcow</code>.'''

<span id="step-2-prepare-system-for-bitwarden-installation"></span>
== Step 2: Prepare system for Bitwarden installation: ==

<span id="ssh-into-the-mailserver-computer"></span>
==== 2.0 SSH into the mailserver computer ====

<pre>ssh louis@192.168.5.3</pre>
OR

<pre>ssh louis@bitwarden.home.arpa</pre>
<span id="update-and-upgrade-your-system-2"></span>
==== 2.1 Update and upgrade your system ====

<pre>sudo apt update &amp;&amp; sudo apt upgrade -y
sudo apt install curl git wget apt-transport-https ca-certificates software-properties-common -y</pre>
<span id="verify-docker-installation"></span>
==== 2.2 Verify Docker installation: ====

<span id="if-you-elected-to-install-mailcow-already-this-part-is-already-done-you-can-skip-to-step-3"></span>
===== IF YOU ELECTED TO INSTALL MAILCOW ALREADY, THIS PART IS ALREADY DONE &amp; YOU CAN SKIP TO STEP 3! =====

**If you installed mailcow &amp; followed the instructions for it, you already installed docker properly on this virtual machine, and have no need to do this again. Skip to step 3 if that is the case.

Run <code>docker --version</code> and make sure the version is 24.0.0 or later. If not, remove the old version:

<pre>sudo apt remove docker docker-engine docker.io containerd runc</pre>
<span id="install-docker-using-official-docker-script-2"></span>
==== 2.3 Install Docker using official Docker script: ====

<pre>curl -fsSL https://get.docker.com -o get-docker.sh
sudo sh get-docker.sh</pre>
<blockquote>'''Note:''' It’s very important to use the official Docker installation and not the Snap version. The Snap version can cause issues due to its sandboxed nature, making it a mess for <code>mailcow</code>’s requirements. It is bad for our purposes, don’t use it.
</blockquote>
<span id="install-docker-compose-prerequisites"></span>
==== 2.4 Install Docker Compose &amp; prerequisites: ====

<pre>sudo apt install docker-compose-plugin -y
sudo systemctl enable --now docker</pre>
<span id="make-sure-it-worked"></span>
==== 2.5 Make sure it worked ====

Run <code>docker compose version</code> and make sure the version is 2.0 or higher.

<span id="step-3-configure-bitwarden-environment"></span>
== Step 3: Configure Bitwarden Environment ==

''[https://bitwarden.com/help/install-on-premise-linux/#install-docker-and-docker-compose Bitwarden’s installation instructions]'' are the opposite of Onlyoffice’s. They actually work, and their documentation is amazing. You can find them ''[https://bitwarden.com/help/install-on-premise-linux/#install-docker-and-docker-compose here]''.

<span id="create-bitwarden-user-and-set-permissions"></span>
==== 3.1 Create Bitwarden user and set permissions ====

<pre>sudo adduser bitwarden
sudo usermod -aG docker bitwarden</pre>
Use the following command to log in as the new user, <code>bitwarden</code>:

<pre>sudo login</pre>
Enter credentials for the <code>bitwarden</code> user to log in.

<span id="create-and-configure-bitwarden-directory"></span>
==== 3.2: Create and Configure Bitwarden Directory ====

<pre>sudo mkdir /opt/bitwarden
sudo chmod -R 700 /opt/bitwarden
sudo chown -R bitwarden:bitwarden /opt/bitwarden</pre>
<span id="enable-docker-service"></span>
==== 3.3: Enable Docker Service ====

<pre>sudo systemctl start docker
sudo systemctl enable docker</pre>
<span id="download-and-prepare-installation-script"></span>
==== 3.4: Download and Prepare Installation Script ====

<pre>cd /opt/bitwarden
curl -Lso bitwarden.sh &quot;https://func.bitwarden.com/api/dl/?app=self-host&amp;platform=linux&quot;
chmod +x bitwarden.sh</pre>
<span id="run-the-installation-script"></span>
==== 3.5: Run the Installation Script ====

<pre>./bitwarden.sh install</pre>
<div class="figure">

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:image-20241107020120196.png
</gallery>

</div>
<div class="figure">

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:image-20241107020217334.png
</gallery>

</div>
<span id="installation-configuration-notes"></span>
==== 3.6 Installation Configuration Notes ====

During installation, you’ll need to provide:

* '''Domain Name''': Use <code>bitwarden.home.arpa</code>
* '''SSL Certificate''': Choose ‘n’ for Let’s Encrypt if using a self-signed certificate
** Bitwarden auto-generates a self-signed certificate for you. Isn’t Bitwarden nice?
* '''Installation Credentials''': Get these from [https://bitwarden.com/host/ bitwarden.com/host]

'''Important''': Your installation ID and key will look similar to:

<pre>462b197d-14f0-410e-a2c6-b21200fd09f2
Pcf8vNk5udgT3dI9OWJj</pre>
<span id="port-configuration"></span>
==== 3.7 Port Configuration ====

If running multiple services (like mailcow), you’ll need to modify the ports in <code>/opt/bitwarden/bwdata/config.yml</code>:

<pre>http_port: 81    # Changed from 80
https_port: 444  # Changed from 443</pre>
<span id="step-4-configure-bitwarden-settings"></span>
== Step 4: Configure Bitwarden Settings ==

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:lu67917r1ezu_tmp_923a1598.png
File:lu67917r1ezu_tmp_f60bd933.png
File:lu67917r1ezu_tmp_4581e09b.png
</gallery>

<span id="set-up-domain-and-email-settings"></span>
==== 4.1: Set Up Domain and Email Settings ====

Edit the environment file:

<pre>nano /opt/bitwarden/bwdata/env/global.override.env</pre>
Add the following configurations. Use the credentials from your Postmark SMTP relay account to fill in the username, password, <code>globalSettings__mail__smtp__port</code>, and <code>globalSettings__mail__smtp__host</code> below. Feel free to adjust them based on your email and who you are using for SMTP relay. '''This assumes that you set up Postmark as an SMTP relay in the mailcow/mailserver section of this guide! If you did not, you will have to find another SMTP relay service'''; Gmail offers one. This is needed so that your Bitwarden instance can send emails to you without them going straight to spam.

<pre>globalSettings__domain__baseUrl=https://bitwarden.home.arpa
globalSettings__mail__smtp__host=smtp.postmarkapp.com
globalSettings__mail__smtp__port=587
globalSettings__mail__smtp__ssl=false
globalSettings__mail__smtp__username=&lt;your_email_username&gt;
globalSettings__mail__smtp__password=&lt;your_email_password&gt;
globalSettings__mail__smtp__from=putfromemailhere@youremail.com
adminSettings__admins=putanadminemailhere@youremail.com</pre>
<span id="apply-changes-and-start-service"></span>
==== 4.2 Apply changes and start service ====

<pre>./bitwarden.sh rebuild
./bitwarden.sh start</pre>
<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:lu67917r1ezu_tmp_99b3c521.png
File:lu67917r1ezu_tmp_9005c753.png
File:lu67917r1ezu_tmp_7df1fc57.png
File:lu67917r1ezu_tmp_4832b6bd.png
File:lu67917r1ezu_tmp_8d7c905.png
File:lu67917r1ezu_tmp_e7d3966a.png
File:lu67917r1ezu_tmp_bc67ac1b.png
File:lu67917r1ezu_tmp_a557cbd9.png
File:lu67917r1ezu_tmp_bda10420.png
File:lu67917r1ezu_tmp_d6aa6557.png
File:lu67917r1ezu_tmp_eb377cee.png
File:lu67917r1ezu_tmp_6543bc05.png
File:lu67917r1ezu_tmp_cb1afd48.png
File:lu67917r1ezu_tmp_235ea9d2.png
File:lu67917r1ezu_tmp_323c7ce.png
</gallery>

<span id="step-5-browser-extension-setup"></span>
== Step 5: Browser Extension Setup ==

# '''Make sure VPN Connection''': Connect to your home server VPN
# '''Install Extension''':
#* [https://chrome.google.com/webstore/detail/bitwarden-free-password-ma/nngceckbapebfimnlniiiahkandclblb Chrome Web Store]
#* Firefox: [https://addons.mozilla.org/en-US/firefox/addon/bitwarden-password-manager/ Firefox Add-ons]

<blockquote>'''Critical Step''': When logging in, change the server URL from bitwarden.com to your self-hosted instance (e.g., <code>https://bitwarden.home.arpa:444</code>) '''DON’T FORGET THE ALTERNATIVE PORT AT THE END IF YOU CHOSE AN ALTERNATIVE PORT!'''
</blockquote>
<span id="optional-pin-extension"></span>
==== Optional: Pin Extension ====

* For Chrome/Brave: Right-click the Bitwarden icon and select “Pin”

<span id="setting-up-zfs-for-data-storage"></span>
= Setting up ZFS for data storage =

<span id="how-were-storing-our-data"></span>
== How we’re storing our data: ==

We’re not keeping your 40 terabytes of GNU/Linux ISOs on solid state storage. That is a waste of money &amp; resources (unless you’re insanely rich). I set up the system drives on SSDs so that my photos, documents, mail, and android backups would be quickly accessible and these services highly responsive. I don’t need that level of responsiveness for my collection of GNU/Linux ISOs, though. This is where ZFS pools come into play.

<span id="what-is-zfs"></span>
=== What is ZFS? ===

ZFS is a complete storage management system that combines:
* File system functionality
* Volume management
* RAID capabilities
* Data integrity checking
* Automatic repair features

It’s like having a RAID controller, Linux LVM, and a file system all in one.

<span id="why-zfs"></span>

=== Why ZFS? ===

<span id="data-integrity-built-in"></span>
==== 1. Data Integrity Built-In ====

* ZFS constantly checks for corruption using checksums
* ZFS automatically repairs corrupted files if you have redundancy
* ZFS saved me twice from the consequences of my bad decisions when I bought Seagate products.

<span id="snapshots-that-actually-work-although-im-not-getting-into-that-here"></span>
==== 2. Snapshots That Actually Work (although I’m not getting into that here) ====

* Take instant snapshots that don’t eat up space
* Roll back changes when you inevitably mess something up
* Keep multiple versions of files without doubling storage needs

<span id="dynamic-stripe-sizes"></span>
==== 3. Dynamic Stripe Sizes ====

* Unlike hardware RAID, ZFS can adjust stripe size on the fly

<span id="zfs-encryption"></span>
== ZFS Encryption: ==

<span id="setting-up-encryption"></span>
=== Setting Up Encryption ===

You have two choices:

# '''Pool-wide encryption''':
#* Everything in the pool is encrypted, or
# '''Dataset-level encryption''':
#* Encrypt only specific datasets
#* Different keys for different datasets
#* More confusing, not necessary IMO here.

<blockquote>'''NOTE''': If you’re encrypting a pool for home use, pool-wide encryption is usually the way to go. Keep it simple unless you have a specific reason not to.
</blockquote>
<span id="whats-a-zfs-pool"></span>
== What’s a ZFS Pool? ==

* Traditional setup: Disk → Partition → Filesystem
* ZFS setup: Disks → Pool → Datasets

The pool:
* Manages all your physical drives
* Handles redundancy (like RAID)
* Provides a storage “pool” that datasets can use

It’s like having a fish pond (the pool) that different fish (datasets) can draw from, rather than a different water tank for each koi fishy.

<span id="understanding-zfs-redundancy"></span>

== Understanding ZFS Redundancy ==

ZFS has built-in redundancy options that are similar to RAID but better implemented. Here are the main types. You choose what works for you:

<span id="mirror-similar-to-raid-1"></span>
=== Mirror (Similar to RAID 1) ===

<pre>Disk 1 ───┐
         ├── Identical copies
Disk 2 ───┘</pre>
* Writes data to multiple disks
* Can lose any disk and still work
* 50% storage efficiency (2 drives = 1 drive’s worth of storage)

<span id="raidz1-similar-to-raid-5"></span>
=== RAIDZ1 (Similar to RAID 5) ===

<pre>Disk 1 ───┐
Disk 2 ───┼── Distributed data + parity
Disk 3 ───┘</pre>
* Can lose one drive
* ~67-75% storage efficiency
* Minimum 3 drives needed

<span id="raidz2-similar-to-raid-6"></span>
=== RAIDZ2 (Similar to RAID 6) ===

<pre>Disk 1 ───┐
Disk 2 ───┤
Disk 3 ───┼── Distributed data + double parity
Disk 4 ───┤
Disk 5 ───┘</pre>
* Can lose ANY two drives
* ~60-80% storage efficiency
* Minimum 4 drives needed

<span id="key-differences-from-hardware-raid"></span>
=== Key Differences from Hardware RAID: ===

# '''No RAID controller needed'''
# '''Self-healing'''
#* Detects &amp; fixes corruption automatically
#* Hardware RAID only handles drive failures though.
#* ZFS handles drive failures AND data corruption!

<blockquote>'''HINT''': '''ZFS IS NOT A BACKUP!''' ZFS redundancy protects against drive failures, but it’s NOT a backup. If you accidentally delete a file or your server dies in a fire, redundancy won’t help you. This is PART of a proper backup solution, it is not in &amp; of itself THE backup solution! Always have proper backups!
</blockquote>
<span id="step-1-choose-hard-drives-that-wont-send-you-to-rossmann-data-recovery-using-backblaze-data"></span>
== Step 1: Choose Hard Drives That Won’t Send you to [https://rossmanngroup.com/hard-drive-data-recovery-service/ Rossmann Data Recovery] using [https://www.backblaze.com/cloud-storage/resources/hard-drive-test-data Backblaze Data] ==

If you spend nine hours setting this server up only to put your data on a Seagate rosewood, I will come through your television like Samara from the ring and pull you down a well. You could either

<ol style="list-style-type: lower-alpha;">
<li><p>trust [https://www.youtube.com/watch?v=qZCMislL6_I&t=49s amazon reviews].</p></li>
<li><p>use data from a company that runs over 260,000 hard drives &amp; publishes their failure rates quarterly</p></li>
<li><p>Use a seagate EXOS or rosewood</p></li></ol>

In order of bad ideas, C, A, then B. We will be doing B.

<span id="find-backblazes-drive-stats-here"></span>
=== Find Backblaze’s Drive Stats [https://www.backblaze.com/cloud-storage/resources/hard-drive-test-data here] ===

When Backblaze publishes failure rates, they’re telling you what drives cost them money to replace. They don’t care which manufacturer looks good. They are honest about which drives are trash, they run them 24/7 in actual mission-critical server environments.

<span id="tips-for-reading-their-reports"></span>
=== Tips for reading their reports: ===

When you look at their quarterly reports, focus on:

# '''Annualized Failure Rate (AFR)'''
#* Under 1% = Great
#* 1-2% = Acceptable
#* Over 2% = No.
#* Over 3% = Probably a seagate rosewood or grenada, you might as well be giving your data to a [https://www.youtube.com/watch?v=qFVwQCFhKSE NYS tax collector]
# '''Drive Age &amp; Sample Size'''
#* A 0% failure rate is useless if they only have 10 drives, Look for models with 1,000+ samples

<div class="figure">

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:lu67917r1ezu_tmp_a8d16e37.png
</gallery>

</div>
<div class="figure">

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:lu67917r1ezu_tmp_5c0f8fea.png
</gallery>

</div>
* Pay attention to how long they’ve been using the drive you’re looking at.

'''Remember: The goal isn’t to spend five hours figuring out what drives are the best, it’s to spend a few minutes to learn which are the worst. A 0.32% vs 0.34% failure rate difference doesn’t matter, a 0.32% to 3.2% difference is what we’re looking to avoid.'''

<span id="step-1.5-label-your-drive-bays-as-you-plug-them-in."></span>
== Step 1.5: Label your drive bays as you plug them in. ==

I like to put the serial number of the drive on my bays, or if not possible to do this without blocking airflow, on the bottom or top of the case in-line with the drive bay. This way if I need to take a drive out I don’t have to guess which is which.

The ''[https://www.rosewill.com/rosewill-rsv-l4412u-black/p/9SIA072GJ92847?seoLink=server-components&seoName=Server%20Chassis Rosewill RSV-L4412U server case]'' is a very nice case for this purpose.

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:lu67917r1ezu_tmp_49d72764.png
File:lu67917r1ezu_tmp_1f3f2e5c.png
</gallery>

<span id="step-2-installing-zfs-on-ubuntu-server"></span>
== Step 2: Installing ZFS on Ubuntu Server ==

We are setting up ZFS on our host system that all of our virtual machines are running on, which is <code>happycloud.home.arpa</code> at <code>192.168.5.2</code>.

<span id="update-system-packages"></span>
==== 2.1 Update System Packages ====

First, make sure your system is up to date:

<pre>sudo apt update &amp;&amp; sudo apt upgrade -y</pre>
<span id="install-zfs-drive-monitoring-packages"></span>
==== 2.2 Install ZFS &amp; Drive Monitoring Packages ====

Install the ZFS utilities:

<pre>sudo apt install zfsutils-linux smartmontools -y</pre>
<span id="load-zfs-kernel-module"></span>
==== 2.3 Load ZFS Kernel Module ====

ZFS should load automatically, but make sure it’s loaded:

<pre>lsmod | grep zfs</pre>
If you don’t see output, load it manually:

<pre>sudo modprobe zfs</pre>
<span id="configure-system-for-zfs"></span>
==== 2.4 Configure System for ZFS ====

'''Adjust ARC (Adaptive Replacement Cache) Size:'''

Create a new sysctl configuration file:

<pre>sudo nano /etc/sysctl.d/10-zfs.conf</pre>
Add these lines to limit ZFS memory usage to 50% of RAM:

<pre># ZFS Maximum ARC Size (50% of RAM)
vm.swappiness=1
vm.min_free_kbytes=1524288
vm.watermark_scale_factor=200</pre>
<span id="apply-sysctl-settings"></span>
==== 2.5 Apply Sysctl Settings ====

<pre>sudo sysctl -p /etc/sysctl.d/10-zfs.conf</pre>
<span id="set-up-automatic-module-loading"></span>
==== 2.6. Set Up Automatic Module Loading ====

Create a new file to make sure ZFS loads at boot:

<pre>sudo nano /etc/modules-load.d/zfs.conf</pre>
Add this line:

<pre>zfs</pre>
<span id="make-sure-install-worked"></span>
==== 2.7 Make Sure Install Worked ====

Run a quick check of ZFS commands:

<pre># Check ZFS command availability
zfs list
zpool list

# Both commands should work (though they'll show no pools yet)</pre>
<span id="best-practices"></span>
=== Best Practices: ===

* Set <code>vm.swappiness=1</code> (use swap only when necessary)
* Keep around 1 gigabyte of RAM per 1TB storage for basic usage
* Use separate boot drive from ZFS pool
* Set up notifications if something dies (we’ll cover this later)
* Plan regular scrub schedule

<span id="step-3-identify-your-hard-drives-in-ubuntu-server"></span>
== Step 3: Identify Your Hard Drives in Ubuntu Server ==

<span id="quick-commands-to-list-drives"></span>
=== Quick Commands to List Drives ===

<span id="list-basic-drive-info"></span>
==== 3.1 List Basic Drive Info ====

<pre>lsblk</pre>
Example output:

<pre>louis@happycloud:~$ lsblk
NAME                             MAJ:MIN RM   SIZE RO TYPE  MOUNTPOINTS
sda                                8:0    0 232.9G  0 disk  
├─sda1                             8:1    0   512M  0 part  
├─sda2                             8:2    0     1G  0 part  
│ └─md127                          9:127  0  1022M  0 raid1 /boot
└─sda3                             8:3    0 231.4G  0 part  
  └─md126                          9:126  0 231.3G  0 raid1 
    └─dm_crypt-0                 252:0    0 231.2G  0 crypt 
      └─ubuntuinstall-ubunturoot 252:1    0 231.2G  0 lvm   /
sdb                                8:16   0   7.3T  0 disk  
sdc                                8:32   0 232.9G  0 disk  
├─sdc1                             8:33   0   512M  0 part  /boot/efi
├─sdc2                             8:34   0     1G  0 part  
│ └─md127                          9:127  0  1022M  0 raid1 /boot
└─sdc3                             8:35   0 231.4G  0 part  
  └─md126                          9:126  0 231.3G  0 raid1 
    └─dm_crypt-0                 252:0    0 231.2G  0 crypt 
      └─ubuntuinstall-ubunturoot 252:1    0 231.2G  0 lvm   /
sdd                                8:48   0   7.3T  0 disk  
sde                                8:64   0   7.3T  0 disk  
sdf                                8:80   0   7.3T  0 disk  
sdg                                8:96   0   7.3T  0 disk  
sdh                                8:112  0   7.3T  0 disk  </pre>
<span id="show-more-detailed-info-including-serial-numbers"></span>
==== 3.2 Show More Detailed Info (including serial numbers) ====

<pre>lsblk -o NAME,SIZE,MODEL,SERIAL</pre>
Example output:

<pre>louis@happycloud:~$ lsblk -o NAME,SIZE,MODEL,SERIAL
NAME                               SIZE MODEL            SERIAL
sda                              232.9G Samsung SSD 870  S61VNJ0R413909T
├─sda1                             512M                  
├─sda2                               1G                  
│ └─md127                         1022M                  
└─sda3                           231.4G                  
  └─md126                        231.3G                  
    └─dm_crypt-0                 231.2G                  
      └─ubuntuinstall-ubunturoot 231.2G                  
sdb                                7.3T ST8000VN004-2M21 WSD5720G
sdc                              232.9G Samsung SSD 870  S61VNG0NC09403N
├─sdc1                             512M                  
├─sdc2                               1G                  
│ └─md127                         1022M                  
└─sdc3                           231.4G                  
  └─md126                        231.3G                  
    └─dm_crypt-0                 231.2G                  
      └─ubuntuinstall-ubunturoot 231.2G                  
sdd                                7.3T ST8000VN004-2M21 WSD5725W
sde                                7.3T WDC WD80EFZX-68U VKJ28YJX
sdf                                7.3T WDC WD80EFZX-68U VKJ02D0X
sdg                                7.3T WDC WD80EFZX-68U VKHZVJ7X
sdh                                7.3T WDC WD80EFZX-68U VKJ1N8KX
louis@happycloud:~$ 
</pre>
<span id="check-drive-health-and-additional-info"></span>
==== 3.3 Check Drive Health and Additional Info ====

<pre>louis@happycloud:~$ sudo smartctl -i /dev/sdd
smartctl 7.4 2023-08-01 r5530 [x86_64-linux-6.8.0-47-generic] (local build)
Copyright (C) 2002-23, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Family:     Seagate IronWolf
Device Model:     ST8000VN004-2M2101
Serial Number:    WSD5725W
LU WWN Device Id: 5 000c50 0e3407989
Firmware Version: SC60
User Capacity:    8,001,563,222,016 bytes [8.00 TB]
Sector Sizes:     512 bytes logical, 4096 bytes physical
Rotation Rate:    7200 rpm
Form Factor:      3.5 inches
Device is:        In smartctl database 7.3/5528
ATA Version is:   ACS-4 (minor revision not indicated)
SATA Version is:  SATA 3.3, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is:    Wed Oct 23 21:10:14 2024 UTC
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

louis@happycloud:~$ sudo smartctl -a /dev/sdd | grep -E 'Command_Timeout|Error_Rate';     echo &quot;&quot;; 
  1 Raw_Read_Error_Rate     0x000f   074   064   044    Pre-fail  Always       -       26263737
  7 Seek_Error_Rate         0x000f   089   060   045    Pre-fail  Always       -       766811756
188 Command_Timeout         0x0032   100   100   000    Old_age   Always       -       0</pre>
<blockquote>'''HINT''': Write down the serial numbers of your drives and which ports they’re connected to. If a drive fails, you’ll want to know exactly which physical drive to replace.
</blockquote>
<span id="understanding-the-output"></span>
==== 3.4 Understanding the Output: ====

* In this case, <code>/dev/sda</code> and <code>/dev/sdc</code> are the two SSDs that comprise the RAID 1 array that Ubuntu Linux Server is installed on.
* <code>sdb</code>, <code>sdd</code>, <code>sde</code>, <code>sdf</code>, and <code>sdg</code> are the hard drives we plugged in.
* The letters go in order of how they’re connected to the motherboard (sometimes).
* Numbers after letters (like <code>sda1</code>) represent partitions

Now you know which drive is which, so let’s set up a ZFS pool.

<span id="step-4-creating-an-encrypted-zfs-pool-with-single-drive-redundancy"></span>
== Step 4: Creating an Encrypted ZFS Pool with Single-Drive Redundancy ==

'''What We’re Setting Up'''

* 6 drives in a RAIDZ2 configuration (similar to RAID6)
* Full encryption with password
* Two drives worth of redundancy
* Ability to survive one drive failure

<span id="verify-our-drives"></span>
=== 4.1 Verify Our Drives ===

First, let’s double-check we’re using the right drives:

<pre>lsblk -o NAME,SIZE,MODEL,SERIAL</pre>
You should see your two operating system drives listed, and the six hard drives we plugged in. Let’s make absolutely sure they’re empty:

<pre># Check if drives have any existing partitions
sudo fdisk -l /dev/sd[bdefgh]</pre>
If you see any partitions, you might want to clear them:

<pre># Only run these if you're SURE these are the right drives
# THIS WILL ERASE ALL DATA ON THESE DRIVES
sudo wipefs -a /dev/sdb
sudo wipefs -a /dev/sdd
sudo wipefs -a /dev/sde
sudo wipefs -a /dev/sdf
sudo wipefs -a /dev/sdg
sudo wipefs -a /dev/sdh</pre>
<span id="create-the-encrypted-pool"></span>
==== 4.2 Create the Encrypted Pool ====

We’ll create a RAIDZ2 pool (similar to RAID6) with encryption:

<pre>sudo zpool create -o ashift=12 
   -O encryption=aes-256-gcm 
   -O keylocation=prompt 
   -O keyformat=passphrase 
   mediapool raidz2 /dev/sdb /dev/sdd /dev/sde /dev/sdf /dev/sdg /dev/sdh</pre>
What do these commands do?

* <code>-o ashift=12</code>: Optimizes for 4K sector drives
* <code>-O encryption=aes-256-gcm</code>: Enables strong encryption
* <code>-O keylocation=prompt</code>: Tells ZFS to ask for password
* <code>-O keyformat=passphrase</code>: Use a password instead of keyfile
* <code>raidz2</code>: Two drive redundancy
* <code>mediapool</code>: Name of your pool (can be whatever you want)

You’ll be prompted for a password. '''USE A STRONG PASSWORD AND DON’T FORGET IT!'''

<span id="set-good-pool-properties"></span>
==== 4.3 Set Good Pool Properties ====

After creation, let’s set some good default properties:

<pre># Enable compression
sudo zfs set compression=lz4 mediapool

# Disable atime updates (better performance)</pre>
<pre>sudo zfs set atime=off mediapool

# Set correct recordsize for general media storage
sudo zfs set recordsize=1M mediapool</pre>
<span id="verify-pool-creation"></span>
==== 4.4 Verify Pool Creation ====

Check that everything is set up correctly:

<pre># Check pool status
sudo zpool status mediapool

# Check pool properties
sudo zpool get all mediapool

# Check encryption is enabled
sudo zfs get encryption mediapool</pre>
The <code>zpool status</code> output should show something like:

<pre>louis@happycloud:~$ sudo zpool status mediapool
  pool: mediapool
 state: ONLINE
config:

    NAME        STATE     READ WRITE CKSUM
    mediapool   ONLINE       0     0     0
      raidz2-0  ONLINE       0     0     0
        sdb     ONLINE       0     0     0
        sdd     ONLINE       0     0     0
        sde     ONLINE       0     0     0
        sdf     ONLINE       0     0     0
        sdg     ONLINE       0     0     0
        sdh     ONLINE       0     0     0

errors: No known data errors</pre>
<span id="create-the-datasets-for-your-data-virtual-machine-backups"></span>
==== 4.5: Create the Datasets for your data &amp; virtual machine Backups ====

Set permissions:

<pre># Set ownership (replace 'louis' with your actual username)
sudo chown louis:louis /mediapool

# Set permissions (only you can access it)
sudo chmod 700 /mediapool</pre>
<span id="test-pool-importexport"></span>
==== 4.6 Test Pool Import/Export ====

Let’s make sure we can properly mount/unmount the pool:

<pre># Export (unmount) the pool
sudo zpool export mediapool

# Import it back
sudo zpool import mediapool</pre>
You’ll have to enter the password with <code>sudo zfs load-key mediapool</code> in order to do anything with it, but we will do that later. You will be prompted for the password again when importing.

<span id="important-notes"></span>
==== Important Notes ====

# '''BACKUP YOUR POOL PASSWORD!'''
#* If you lose it, your data is GONE
#* Store it in a password manager (that you don’t self-host)
#* Consider a paper backup in a secure location that is not a post-it-note on your monitor.
# '''Space Available'''
#* Total raw capacity: 6 × 8TB = 48TB
#* RAIDZ2 uses 2 drives for parity, so you lose 2 drives worth of capacity
#* Usable space is 4 × 8TB = 32TB
# '''What Redundancy Gives You'''
#* Can survive one drive failure
#* Can survive two drive failures
#* '''Not a backup! Still need proper backups'''

<span id="step-5-setting-up-zfs-pool-mount-points-and-permissions"></span>
== Step 5: Setting Up ZFS Pool Mount Points and Permissions ==

<span id="creating-the-base-dataset-structure"></span>
==== 5.1 Creating the Base Dataset Structure ====

First, let’s create our main dataset and its subdirectories:

<pre># Load the encryption key so we can work:
sudo zfs load-key mediapool

# Create mount points if they don't exist

# Create the virtual machine backup dataset where we'll store VM images
sudo zfs create -o mountpoint=/mediapool/vmbackups mediapool/vmbackups

# Create the storage backup dataset where we'll store Linux ISOs and cooking recipes
sudo zfs create -o mountpoint=/mediapool/archive mediapool/archive</pre>
<span id="setting-permissions-for-regular-user-access"></span>
==== 5.2 Setting Permissions for Regular User Access ====

Set ownership for the main archive directory:

<pre># Set ownership of the main archive directory to louis
sudo chown louis:louis /mediapool/archive

# Set base permissions (rwx for owner, rx for group and others)
sudo chmod 755 /mediapool/archive</pre>
<span id="securing-vmbackups-directory-for-root-only"></span>
==== 5.3 Securing vmbackups Directory for Root Only ====

Set restricted permissions on the vmbackups directory:

<pre># Set vmbackups to be owned by root
sudo chown root:root /mediapool/vmbackups

# Set permissions to allow only root access (rwx for root, none for others)
sudo chmod 700 /mediapool/vmbackups</pre>
<span id="verify-the-settings"></span>
==== 5.4 Verify the Settings ====

Check that everything is set correctly:

<pre># Check ZFS mountpoints
zfs get mountpoint mediapool/archive
zfs get mountpoint mediapool/vmbackups

# Check permissions
ls -la /mediapool/archive
ls -la /mediapool/vmbackups

# Verify dataset properties
zfs get all mediapool/archive
zfs get all mediapool/vmbackups</pre>
Expected output for permissions check, note that user <code>louis</code> cannot list the <code>vmbackups</code> directory without sudo.

<pre>louis@happycloud:~$ zfs get mountpoint mediapool/archive
NAME               PROPERTY    VALUE               SOURCE
mediapool/archive  mountpoint  /mediapool/archive  local

louis@happycloud:~$ zfs get mountpoint mediapool/vmbackups
NAME                 PROPERTY    VALUE                 SOURCE
mediapool/vmbackups  mountpoint  /mediapool/vmbackups  local

louis@happycloud:~$ ls -la /mediapool/archive
total 21
drwxr-xr-x 2 louis louis    2 Oct 23 21:45 .
drwxr-xr-x 4 root  root  4096 Oct 23 21:45 ..

louis@happycloud:~$ ls -la /mediapool/vmbackups
ls: cannot open directory '/mediapool/vmbackups': Permission denied

louis@happycloud:~$ sudo ls -la /mediapool/vmbackups
total 21
drwx------ 2 root root    2 Oct 23 21:44 .
drwxr-xr-x 4 root root 4096 Oct 23 21:45 ..
</pre>
<span id="test-access"></span>
==== 5.5 Test Access ====

Test the permissions are working:

<ol style="list-style-type: decimal;">
<li><p>As user ‘louis’:</p>
<pre># Should work
touch /mediapool/archive/testfile

# Should fail
touch /mediapool/vmbackups/testfile</pre></li>
<li><p>As root:</p>
<pre># Should work
sudo touch /mediapool/vmbackups/testfile</pre></li></ol>

If any of these tests don’t work as expected, double-check the permissions and ownership settings above.

<span id="frigate-camera-footage-storage"></span>
==== 5.6 frigate camera footage storage ====

Earlier in the guide, we set up '''frigate''' for recording security camera footage. We left it recording to the frigate installation folder. '''This is bad. Recording to the main solid state drive is a waste of space &amp; SSD life.'''

Archived camera footage belongs on a giant hard drive, not an expensive SSD. If you’d like, you can now go back to the frigate config section and change these two lines:

<pre>      - ./storage:/media/frigate
      - ./database:/data/db</pre>
to something like:

<pre>      - ./storage:/mediapool/archive/camerafootage/media/frigate
      - ./database:/mediapool/archive/camerafootage/data/db</pre>
Of course, make the directories first:

<pre>mkdir -p /mediapool/archive/camerafootage/data/db
mkdir -p /mediapool/archive/camerafootage/media/frigate</pre>
If you want to keep things separate, you could create a third dataset called <code>camerafootage</code>, mount it to <code>/mediapool/camerafootage</code>, and then edit the <code>docker-compose.yml</code> file to look like this:

<pre>      - ./storage:/mediapool/camerafootage/media/frigate
      - ./database:/mediapool/camerafootage/data/db</pre>
And make sure the directories have been created before running frigate:

<pre>mkdir -p /mediapool/camerafootage/data/db
mkdir -p /mediapool/camerafootage/media/frigate</pre>
The full file is provided below, with the assumption that you decided to make a <code>camerafootage</code> dataset that is mounted on <code>/mediapool/camerafootage</code>

<pre>version: &quot;3.9&quot;
services:
  frigate:
    container_name: frigate
    privileged: true # This may not be necessary for all setups
    restart: unless-stopped
    image: ghcr.io/blakeblackshear/frigate:0.13.2 # Last good version
    shm_size: &quot;64mb&quot; # Update for your cameras based on requirements
    devices:
      - /dev/bus/usb:/dev/bus/usb # USB Coral, modify for other hardware
      - /dev/apex_0:/dev/apex_0 # PCIe Coral, modify based on your setup
      - /dev/video11:/dev/video11 # For Raspberry Pi 4B
      - /dev/dri/renderD128:/dev/dri/renderD128 # Intel hwaccel, update for your hardware
    volumes:
      - /etc/localtime:/etc/localtime:ro
      - ./config:/config
      - ./storage:/mediapool/camerafootage/media/frigate # Changed media directory to ZFS pool
      - ./database:/mediapool/camerafootage/data/db # Changed database directory to ZFS pool
      - type: tmpfs # Optional: Reduces SSD wear
        target: /tmp/cache
        tmpfs:
          size: 1000000000
    ports:
      - &quot;8971:8971&quot;
      - &quot;5000:5000&quot; # Internal unauthenticated access. Be careful with exposure.
      - &quot;8554:8554&quot; # RTSP feeds
      - &quot;8555:8555/tcp&quot; # WebRTC over TCP
      - &quot;8555:8555/udp&quot; # WebRTC over UDP
    environment:
      FRIGATE_RTSP_PASSWORD: &quot;password&quot;</pre>
<span id="step-6-setting-up-samba-to-share-zfs-pool-directories"></span>
== Step 6: Setting Up Samba to Share ZFS Pool Directories ==

<span id="installing-samba"></span>
==== 6.1 Installing Samba ====

First, let’s install Samba and its utilities:

<pre># Update package list
sudo apt update

# Install Samba packages
sudo apt install samba samba-common-bin -y</pre>
<span id="backup-original-samba-config"></span>
==== 6.2 Backup Original Samba Config ====

Always backup before making changes:

<pre>sudo cp /etc/samba/smb.conf /etc/samba/smb.conf.backup</pre>
<span id="configure-samba-share"></span>
==== 6.3 Configure Samba Share ====

Create a new Samba configuration:

<pre># Clear existing config (but keep our backup)
sudo bash -c 'echo &quot;&quot; &gt; /etc/samba/smb.conf'

# Edit the config file
sudo nano /etc/samba/smb.conf</pre>
Add this configuration to <code>smb.conf</code>, and change the <code>realm</code> to the domain you chose in <code>pfsense</code> under <code>system ---&gt; general setup</code>

<pre>[global]
    # Network settings
    workgroup = HOME
    realm = home.arpa
    netbios name = happycloud
    server string = ZFS Archive Server
    dns proxy = no
    
    # Security settings
    security = user
    map to guest = bad user
    server signing = auto
    client signing = auto
    
    # Logging
    log level = 1
    log file = /var/log/samba/%m.log
    max log size = 1000
    
    # Performance optimization
    socket options = TCP_NODELAY IPTOS_LOWDELAY
    read raw = yes
    write raw = yes
    use sendfile = yes
    min receivefile size = 16384
    aio read size = 16384
    aio write size = 16384
    
    # Multichannel support
    server multi channel support = yes
    
    # Disable unused services
    load printers = no
    printing = bsd
    printcap name = /dev/null
    disable spoolss = yes
    
    # Character/Unix settings
    unix charset = UTF-8
    dos charset = CP932

[archive]
    comment = ZFS Archive Share
    path = /mediapool/archive
    valid users = louis
    invalid users = root
    browseable = yes
    read only = no
    writable = yes
    create mask = 0644
    force create mode = 0644
    directory mask = 0755
    force directory mode = 0755
    force user = louis
    force group = louis
    veto files = /._*/.DS_Store/.Thumbs.db/.Trashes/
    delete veto files = yes
    follow symlinks = yes
    wide links = yes
    ea support = yes
    inherit acls = yes
    hide unreadable = yes</pre>
<span id="verify-samba-configuration"></span>
==== 6.4 Verify Samba Configuration ====

Check if your config is valid:

<pre>testparm</pre>
<span id="create-samba-user"></span>
==== 6.5 Create Samba User ====

Add your GNU/Linux user to Samba and set a password:

<pre># Add Samba password for user 'louis'
sudo smbpasswd -a louis

# Enable the user
sudo smbpasswd -e louis</pre>
<span id="start-and-enable-samba"></span>
==== 6.6 Start and Enable Samba ====

<pre># Restart Samba services
sudo systemctl restart smbd
sudo systemctl restart nmbd

# Enable them to start at boot
sudo systemctl enable smbd
sudo systemctl enable nmbd</pre>
<span id="step-7-connecting-to-your-samba-share"></span>
== Step 7: Connecting to your Samba Share ==

What’s the point of this if we can’t access it from other systems?

<span id="windows-systems"></span>
==== Windows Systems ====

Connect using one of the following in the address bar of Windows Explorer:

* <code>\\happycloud.home.arpa\archive</code>

<span id="linux-systems"></span>
==== GNU/Linux Systems ====

Connect in a file manager like Thunar (my personal favorite) by putting this in the address bar:

* <code>smb://happycloud.home.arpa/archive</code>

'''File Manager Navigation:'''

# Press <code>Ctrl+L</code> to open location bar
# Enter the SMB URL
# Enter credentials when prompted

<span id="macos-systems"></span>
==== macOS Systems ====

Connect using Finder by selecting <code>Go</code> &gt; <code>Connect to Server</code> and entering the SMB URL.

Connect using:

* <code>smb://happycloud.home.arpa/archive</code>

'''Finder Navigation:'''

# Press <code>Cmd+K</code>
# Enter the SMB URL
# Click ‘Connect’
# Enter credentials when prompted

<span id="mounting-from-command-line-linux"></span>
==== Mounting from Command Line (GNU/Linux) ====

If you want the share to show up as if it were just another directory on your system, you could do this:

<pre># Create mount point
mkdir -p ~/archive

# Mount by entering credentials when prompted
sudo mount -t cifs //happycloud.home.arpa/archive ~/archive -o username=louis,uid=1000,gid=1000,vers=3.1.1,seal

# Check that the `testfile` we made earlier shows up here. If you see the following, congratulations, you did not mess it up!!

[louis@studiobauer ~]$ ls -la ~/archive
total 13
drwxr-xr-x  2 louis louis     0 Oct 23 18:11 .
drwx------ 48 louis louis 12288 Oct 23 18:14 ..
-rwxr-xr-x  1 louis louis     0 Oct 23 18:11 testfile</pre>
<blockquote>'''HINT''': If you can’t connect via VPN, try from local network first. If that works, then troubleshoot VPN/remote access issues afterwards.
</blockquote>
<span id="security-notes"></span>
== Security Notes ==

# The share is only accessible to authenticated users
# Files created will be owned by ‘louis’
# The VMBackups directory remains inaccessible (root only)
# Password is stored separately from system password
# All traffic is unencrypted - use VPN for remote access!

Now you should be able to access your ZFS pool’s archive directory from any device on your network, with proper authentication as user ‘louis’.

<span id="step-7-backing-up-virtual-machines"></span>
= Step 7: Backing up virtual machines =

Now that we have a giant storage array that will continue working even in the event of multiple drive deaths, we can set up our virtual machines to back up regularly. This way, if we destroy one with idiocy, or if it becomes corrupt, we can restore it instantly to what it was like before the mess happened.

<span id="backup-script-creation"></span>
==== 7.1 Backup script creation: ====

This script below will allow you to have your virtual machines backed up automatically. It does the following:

* Shuts down the virtual machine
* Copies its disk image qcow2 file to the <code>/mediapool/vmbackups</code> zfs dataset
* Copies its configuration so it can be set up again
* Keeps five backups but deletes the oldest ones after you have five.

This means the following:

* You can mess things up by deleting files you weren’t supposed to, mess up configurations and programs, and restore everything to where it was last week with one or two kindergarten level GNU/Linux commands.
* You can migrate this to another computer entirely &amp; start the virtual machine up there.

shuts each virtual machine down one by one, backs up the virtual

Save this as <code>/root/vm_backup.sh</code>:

<pre># Open the text editor
sudo nano -w /root/vm_backup.sh</pre>
<pre>#!/bin/bash

# thank you to stack overflow for giving me the courage to wade through 100s of posts and hack together something that looks like it works. 

# config for backups
BACKUP_DIR=&quot;/mediapool/vmbackups&quot;
LOG_FILE=&quot;/var/log/vm_backups.log&quot;
RETENTION_DAYS=56  # how long to keep backups

# Function to write messages to our log file`
log_message() {
    # Get the current timestamp and message
    echo &quot;[$(date '+%Y-%m-%d %H:%M:%S')] $1&quot; | tee -a &quot;$LOG_FILE&quot;
}

# Function to find the actual disk path for a VM when the default path doesn't exist
# Uses virsh dumpxml to get the disk source path from the VM's XML configuration
find_vm_disk_path() {
    local vm_name=$1
    # Get the VM's XML configuration and extract the first disk source path
    # Using grep with -o to only output the matched portion
    # Using sed to extract just the path part from the source attribute
    local disk_path=$(virsh dumpxml &quot;$vm_name&quot; | grep -o &quot;source file='[^']*'&quot; | head -n1 | sed &quot;s/source file='\(.*\)'/\1/&quot;)
    
    # Check if we found a path and if it exists
    if [ -n &quot;$disk_path&quot; ] &amp;&amp; [ -f &quot;$disk_path&quot; ]; then
        echo &quot;$disk_path&quot;
        return 0
    else
        return 1
    fi
}

# main backup function 
backup_vm() {
    local virtual_machine_name=$1  # The name of the virtual machine we're backing up
    local date_stamp=$(date +%Y%m%d)  # Today's date for the backup file name
    local source_file=&quot;/var/lib/libvirt/images/${virtual_machine_name}.qcow2&quot;  # Where the virtual machine is
    
    # If the default path doesn't exist, try to find the actual disk path
    if [ ! -f &quot;$source_file&quot; ]; then
        log_message &quot;Default disk path not found for ${virtual_machine_name}, searching XML configuration...&quot;
        local found_path=$(find_vm_disk_path &quot;$virtual_machine_name&quot;)
        
        # If we found a valid path, use it instead
        if [ -n &quot;$found_path&quot; ]; then
            log_message &quot;Found alternate disk path: ${found_path}&quot;
            source_file=&quot;$found_path&quot;
        fi
    fi
    
    local backup_file=&quot;${BACKUP_DIR}/${virtual_machine_name}-${date_stamp}.qcow2&quot;  # Where we're putting the backup of it
    local config_file=&quot;${BACKUP_DIR}/${virtual_machine_name}-${date_stamp}.xml&quot;  # Where it saves the virtual machine config
    
    # Check if source file exists before attempting backup
    if [ ! -f &quot;$source_file&quot; ]; then
        log_message &quot;ERROR: Source file $source_file does not exist for ${virtual_machine_name}&quot;
        return 1
    fi
    
    # Announce backup is starting
    log_message &quot;Starting backup process for ${virtual_machine_name}&quot;
    
    # Save virtual machine's config
    virsh dumpxml &quot;$virtual_machine_name&quot; &gt; &quot;$config_file&quot;
    
    # Set ownership and permissions for config file
    chown libvirt-qemu:kvm &quot;$config_file&quot;
    chmod 644 &quot;$config_file&quot;

    # Try to shut down the virtual machine nicely 
    log_message &quot;Shutting down ${virtual_machine_name}&quot;
    virsh shutdown &quot;$virtual_machine_name&quot;
    
    # Wait patiently for the virtual machine to shut down 
    local count=0
    while [ &quot;$(virsh domstate $virtual_machine_name)&quot; != &quot;shut off&quot; ] &amp;&amp; [ $count -lt 30 ]; do
        sleep 10
        count=$((count + 1))
    done
    
    # If it doesn't turn off, make it turn off(like holding the power button)
    if [ &quot;$(virsh domstate $virtual_machine_name)&quot; != &quot;shut off&quot; ]; then
        log_message &quot;WARNING: Force shutting down ${virtual_machine_name}&quot;
        virsh destroy &quot;$virtual_machine_name&quot;
        sleep 10
    fi
    
    # Make sure it's actually off - trust but verify
    if [ &quot;$(virsh domstate $virtual_machine_name)&quot; != &quot;shut off&quot; ]; then
        log_message &quot;ERROR: Failed to shut down ${virtual_machine_name}&quot;
        return 1
    fi
    
    # Create the backup - doesn't use compression since qemu-img convert compression is single threaded and insanely slow
    log_message &quot;Creating backup of ${virtual_machine_name}&quot;
    if ! qemu-img convert -p -f qcow2 -O qcow2 &quot;$source_file&quot; &quot;$backup_file&quot;; then
        log_message &quot;ERROR: Backup failed for ${virtual_machine_name}&quot;
        virsh start &quot;$virtual_machine_name&quot;
        return 1
    fi
    
    # Set ownership and permissions for backup file
    chown libvirt-qemu:kvm &quot;$backup_file&quot;
    chmod 644 &quot;$backup_file&quot;

    # Make sure the backup isn't insanely small since that means this didn't work
    # Fixed stat command for Linux systems
    local source_size=$(stat -c%s &quot;$source_file&quot;)
    local backup_size=$(stat -c%s &quot;$backup_file&quot;)
    if [ &quot;$backup_size&quot; -lt 1048576 ]; then  # Less than 1MB is suspicious - like a $5 &quot;genuine&quot; Rolex
        log_message &quot;ERROR: Backup file suspiciously small for ${virtual_machine_name}&quot;
        rm -f &quot;$backup_file&quot; &quot;$config_file&quot;
        virsh start &quot;$virtual_machine_name&quot;
        return 1
    fi
    
    # Turn virtual machine back on when backup is done. 
    log_message &quot;Starting ${virtual_machine_name}&quot;
    virsh start &quot;$virtual_machine_name&quot;
    
    # Wait for it to come back online 
    count=0
    while [ &quot;$(virsh domstate $virtual_machine_name)&quot; != &quot;running&quot; ] &amp;&amp; [ $count -lt 12 ]; do
        sleep 5
        count=$((count + 1))
    done
    
    # Make sure it actually started(inspect what you expect)
    if [ &quot;$(virsh domstate $virtual_machine_name)&quot; != &quot;running&quot; ]; then
        log_message &quot;ERROR: Failed to start ${virtual_machine_name}&quot;
        return 1
    fi
    
    # announce that it worked
    log_message &quot;Backup of ${virtual_machine_name} completed!&quot;
    
    # Clean up old backups - because nobody likes a full hard drive
    log_message &quot;Cleaning up old backups for ${virtual_machine_name}&quot;
    find &quot;$BACKUP_DIR&quot; -name &quot;${virtual_machine_name}-*.qcow2&quot; -mtime +${RETENTION_DAYS} -exec rm -f {} \;  # Delete old qcow2 files
    find &quot;$BACKUP_DIR&quot; -name &quot;${virtual_machine_name}-*.xml&quot; -mtime +${RETENTION_DAYS} -exec rm -f {} \;   # Delete old xml files
}

# Start of the main backup process
log_message &quot;Starting backup process&quot;

# Make sure we're running as root 
if [ &quot;$EUID&quot; -ne 0 ]; then
    log_message &quot;ERROR: Must run as root&quot;
    exit 1
fi

# Check if the backup directory exists 
if [ ! -d &quot;$BACKUP_DIR&quot; ]; then
    log_message &quot;ERROR: Backup directory $BACKUP_DIR does not exist&quot;
    exit 1
fi

# Get list of ALL virtual machines, not just running ones
# Changed to list all VMs instead of just running ones
VMS=($(virsh list --all --name))

# Check if we have enough disk space to back up
available_space=$(df -B1 &quot;$BACKUP_DIR&quot; | awk 'NR==2 {print $4}')
required_space=0

# Calculate how much space we need
for virtual_machine in &quot;${VMS[@]}&quot;; do
    if [ -n &quot;$virtual_machine&quot; ]; then
        # Try the default path first
        local_path=&quot;/var/lib/libvirt/images/${virtual_machine}.qcow2&quot;
        
        # If default path doesn't exist, try to find actual path
        if [ ! -f &quot;$local_path&quot; ]; then
            local_path=$(find_vm_disk_path &quot;$virtual_machine&quot;) || continue
        fi
        
        if [ -f &quot;$local_path&quot; ]; then
            virtual_machine_size=$(du -b &quot;$local_path&quot; 2&gt;/dev/null | cut -f1)
            required_space=$((required_space + virtual_machine_size))
        fi
    fi
done

# Make sure we have enough space 
if [ &quot;$available_space&quot; -lt &quot;$required_space&quot; ]; then
    log_message &quot;ERROR: Insufficient space in backup directory&quot;
    exit 1
fi

# loop for backing up every virtual machine
for virtual_machine in &quot;${VMS[@]}&quot;; do
    if [ -n &quot;$virtual_machine&quot; ]; then
        backup_vm &quot;$virtual_machine&quot;
    fi
done

# announce it's all done
log_message &quot;Backup process completed!&quot;
</pre>
<blockquote>'''Nerd note: This script would be laughed out of the room for use in production environments for major web companies &amp; datacenters.''' This script turns off the virtual machine to back it up.

This means that at 1 AM, the service goes down. This would be unacceptable in a production environment where people expect the service to be available 24/7.

There are ways to do live backups where you flush mysql tables and lock them, make redis background save, pause call processing in asterisk, pause io, create atomic snapshots, coordinate with databases of all the different programs…. the audience of this guide is a person running a home server in his closet. Do you really want to subject a beginner to docker volumes that may not be in a consistent state, email delivery/receipt being interrupted, database transactionst hat are messed with in the middle fo a write, corrupt call recordings, partially written large files, all so someone can get live backups of a server in their closet, you monster?

If you need that level of uptime, you’re not a newbie reading this guide. or you are, and you need to hire a consultant to set you up with something like veeam.

To subject a newbie to the risk of error/corruption/screwups that comes with doing live backups for these things when they’re at the level of this guide being helpful to them is cruel.
</blockquote>
<span id="set-permissions-so-script-works"></span>
==== 7.2 Set permissions so script works ====

This won’t work if we don’t give it permissions to be executable.

<pre># Make script executable
sudo chmod +x /root/vm_backup.sh

# Test script permissions
sudo -u root /root/vm_backup.sh</pre>
<span id="tell-computer-to-run-script-every-week-at-1-am-on-sunday"></span>
==== 7.3 Tell computer to run script every week, at 1 AM on Sunday ====

Cron is a scheduler. You can tell cron to run a command, a script, etc. once a week, once a month, twice a day, every 10 minutes. We’re going to set this to back up at 1 AM every Sunday.

<ol style="list-style-type: decimal;">
<li><p>Open root’s crontab:</p>
<pre>sudo crontab -e</pre></li>
<li><p>Add this line:</p>
<pre>0 1 * * 0 /root/vm_backup.sh &gt;&gt; /var/log/vm_backup.log 2&gt;&amp;1</pre>
<p>This will:</p>
<ul>
<li><p>Run at 1:00 AM every Sunday</p></li>
<li><p>Log output to <code>/var/log/vm_backup.log</code></p></li>
<li><p>Include both standard output and errors in the log</p></li>
<li><p>The virtual machine will be down while the transfer occurs</p></li></ul>

<p>If anyone is calling Rossmann Repair Group at 1 AM on a Sunday morning, they deserve to get a busy signal. Actually they deserve allison smith telling them to get the fk out of here. but a busy signal will suffice.</p></li></ol>

==== 7.4 Make sure cron is running ====

<pre>sudo systemctl status cron</pre>
View scheduled cron jobs:

<pre>sudo crontab -l</pre>
<span id="step-8-restoring-a-virtual-machine-from-a-backup"></span>

== Step 8: Restoring a virtual machine from a backup ==

So you messed up and deleted everything inside your virtual machine. You want to go back to where you were before.

Remember:

'''A BACKUP PLAN IS ONLY AS GOOD AS HOW EASY IT IS TO RESTORE FROM A BACKUP!'''

<span id="basic-restore"></span>
=== Basic Restore ===

By “basic restore” I mean what to do when you messed up a program configuration or deleted files inside a virtual machine or corrupted something accidentally. You want to go back to the image of the virtual machine you had before, on the same happycloud host computer.

<span id="before-you-start"></span>
==== 8.1 Before You Start ====

I’m assuming the following is true: - Your virtual machine is already defined in Virtual Machine Manager(you see it when you run virtual machine manager GUI) - Your backups are in <code>/mediapool/vmbackups</code> - The backups were created using the qemu-img backup script I provided above - You just need to restore the virtual machine’s disk because you messed up some files or programs

<span id="find-your-backup"></span>
==== 8.2 Find Your Backup ====

# List available backups for your virtual machine:

<pre>ls -l /mediapool/vmbackups/name-of-your-virtual-machine-*.qcow2</pre>
You’ll see files named like this: - <code>name-of-your-virtual-machine-20240101.qcow2</code> - <code>name-of-your-virtual-machine-20240108.qcow2</code>

These are the disk image files that have all of the data/programs/databases/operating system.

Each backup will have an XML file to go with it: - <code>name-of-your-virtual-machine-20240101.xml</code> - <code>name-of-your-virtual-machine-20240108.xml</code>

These are the files that tell virtual machine manager all of the details about your virtual machine(RAM/CPU, hardware setup, etc.)

Pick the most recent backup before you screwed something up.

<span id="fast-restore"></span>
==== Fast Restore: ====

# Turn off the virtual machine

<pre># Shut down the virtual machine gracefully
virsh shutdown name-of-your-virtual-machine

# Wait until it's actually off. Check status with:
virsh list --all</pre>
<ol start="2" style="list-style-type: decimal;">
<li>Backup Current Disk (just in case)</li></ol>

<pre># Move the current (messed up/broken) disk with date
mv /var/lib/libvirt/images/name-of-your-virtual-machine.qcow2 /var/lib/libvirt/images/name-of-your-virtual-machine.qcow2.broken-$(date +%Y%m%d)</pre>
<ol start="3" style="list-style-type: decimal;">
<li>Restore Backup</li></ol>

<pre># a cool command to put the virtual machine back where it was
qemu-img convert -p -f qcow2 -O qcow2 /mediapool/vmbackups/name-of-your-virtual-machine-20240101.qcow2 /var/lib/libvirt/images/name-of-your-virtual-machine.qcow2

# set permissions so that our virtual machine management stuff can use it.
chown libvirt-qemu:kvm /var/lib/libvirt/images/name-of-your-virtual-machine.qcow2
chmod 644 /var/lib/libvirt/images/name-of-your-virtual-machine.qcow2</pre>
<ol start="4" style="list-style-type: decimal;">
<li>Start the Virtual Machine</li></ol>

<pre>virsh start name-of-your-virtual-machine</pre>
<span id="check-the-restore"></span>
==== Check the Restore ====

# Watch the virtual machine console in Virtual Machine Manager to make sure it boots
# Try logging in when it’s up
# Check that services(mailcow, immich, syncthing) actually work

<span id="complicated-restore"></span>
=== Complicated Restore ===

Let’s say you destroyed more. You also messed up the virtual machine’s configuration in virsh. You edited the xml file for the virtual machine or messed with its settings in the '''Virtual Machine Manager''' GUI, and now nothing works.

For a complete restore of both disk &amp; configuration:

# Remove the current virtual machine:

<pre>virsh destroy name-of-your-virtual-machine
virsh undefine name-of-your-virtual-machine</pre>
<ol start="2" style="list-style-type: decimal;">
<li>Restore the Disk:</li></ol>

<pre># Convert the compressed backup to the images directory
qemu-img convert -p -f qcow2 -O qcow2 /mediapool/vmbackups/name-of-your-virtual-machine-20240101.qcow2 /var/lib/libvirt/images/name-of-your-virtual-machine.qcow2

# Fix permissions
chown libvirt-qemu:kvm /var/lib/libvirt/images/name-of-your-virtual-machine.qcow2
chmod 644 /var/lib/libvirt/images/name-of-your-virtual-machine.qcow2</pre>
<ol start="3" style="list-style-type: decimal;">
<li>Restore the virtual machine config:</li></ol>

<pre># The backup includes the XML configuration file
virsh define /mediapool/vmbackups/name-of-your-virtual-machine-20240101.xml</pre>
<ol start="4" style="list-style-type: decimal;">
<li>Start the VM:</li></ol>

<pre>virsh start name-of-your-virtual-machine</pre>
<span id="common-screwups"></span>
=== Common screwups ===

# '''“Failed to convert image”''':
#* Make sure you have enough disk space
#* Check that the backup file isn’t corrupted
# '''“Failed to start VM”''':
#* Usually permissions. Everyone is excited to realize they had a backup file whilst copying it back; in the excitement of realizing you actually HAVE a backup, nobody remembers to set permissions on the backup file.
#* Check that the XML file matches the system config. use virtual machine manager for this to see if anything sticks out in the GUI as a stupid mistake.
#* Verify all paths exist
# '''“Could not access storage file”''': Check paths in both:
#* <code>/var/lib/libvirt/images/</code>
#* The virtual machine XML config
#* Make sure permissions are right (644 for files)

<span id="verifying-success"></span>
=== Verifying Success ===

After restoration, verify: 1. VM boots properly 2. Network connectivity works 3. All services start correctly 4. Data and configurations are as expected 5. Check logs for any errors

If something isn’t right, you can always try an older backup - they’re kept for 56 days.

<span id="accessing-your-samba-share-from-any-device"></span>
= Accessing Your Samba Share from Any Device =

Let’s say you want to watch a GNU/Linux ISO while you’re on the go. You connect to your VPN, and you can browse your files right there. OwlFiles can play music &amp; video files right inside the application and stream them without you having to download them, for a wide variety of codecs, and it does so exceptionally well. It even gives options for hardware vs. software decoding of the video file in case one works better than the other for the format you’re using.

It’s not open source, but it’s the best samba client I have ever used for android.

<span id="android-access-with-owlfiles"></span>
== Android Access with OwlFiles ==

<span id="install-owlfiles"></span>
=== 1. Install OwlFiles ===

# Open Google Play Store
# Search for “OwlFiles”
# Install the app (it’s free!)

<span id="configure-owlfiles-for-samba-access"></span>
=== 2. Configure OwlFiles for Samba Access ===

# Open OwlFiles
# Tap the “+” button in the bottom right
# Select “Network Storage (SMB)”
# Fill in the connection details:
#* '''Server''': Your server’s IP (e.g., <code>192.168.5.2</code>)
#* '''Share''': <code>zfsarchive</code>
#* '''Username''': <code>louis</code>
#* '''Password''': Your Samba password
#* '''Name''': Whatever you want to call it (e.g., “Home Server”)
# Tap “Test Connection” to verify
# Tap “Save” if test is successful

<span id="using-owlfiles"></span>
=== 3. Using OwlFiles ===

# '''Browse Files:'''
#* Tap your newly created connection
#* Navigate through folders
#* Files will stream rather than download first
# '''Stream Media:'''
#* Tap a video/audio file to stream
#* No need to download completely first
# '''File Operations:'''
#* Long-press files for options
#* Copy, move, delete as needed
#* Upload from phone to server

<blockquote>'''HINT''': Enable “Show hidden files” in settings if you need to see dot files.
</blockquote>
<span id="file-operation-best-practices"></span>
=== File Operation Best Practices: ===

# '''Large Files:'''
#* Use copy instead of move for safety
#* Don’t interrupt transfers
#* Check free space first
# '''Media Streaming:'''
#* Test a small file first
#* Check your connection speed
#* Consider pre-downloading for trips

* Long-press files for options
* Copy, move, delete as needed
* Upload from phone to server

'''HINT''': Enable “Show hidden files” in settings if you need to see dot files.

= Have your server email you when a hard drive is dying. =

There is one caveat that makes ZFS &amp; RAID functionally ''useless'' for many of its users..

99% of the population don’t know their drive is failing until things start crashing and working horribly slow. By then, it’s usually too late. You’re heading to Rossmann Repair for data recovery.

Then they think, ''“if I use RAID, I’m good! One drive can fail and it’ll still work!!!”''

No.

You could have RAID 1 with 20 discs and it still wouldn’t matter, because ''NOBODY WHO HAS A LIFE CHECKS THE HEALTH OF THEIR DISK DRIVE EVERY DAY.''

If you only check your drive health when it fails, then RAID 1 with 5 disks is useless. You’re still only going to check it when the fifth one starts failing.

<span id="step-1-setting-up-postfix-email-system-on-ubuntu-server-24.04"></span>
== Step 1: Setting Up Postfix Email System on Ubuntu Server 24.04 ==

<span id="install-required-packages"></span>
==== 1.1 Install Required Packages ====

<pre>sudo apt update
sudo apt install postfix libsasl2-modules mailutils -y</pre>
When prompted during install:

* Choose '''“Internet Site”''' for configuration type
* Enter your system’s fully qualified domain name when asked of where we are sending emails from, in our case it is <code>home.arpa</code>
* Recipient for root &amp; postmaster mail will be the email you want to receive that at, for me I set it as the same email as ZFS alerts which is <code>l.a.rossmann@gmail.com</code> for me
* Set '''“Force synchronous updates on mail queue?”''' to no

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:lu67917r1ezu_tmp_ff734222.png
File:lu67917r1ezu_tmp_667e9c06.png
File:lu67917r1ezu_tmp_f9f6cd56.png
File:lu67917r1ezu_tmp_5c8e2e53.png
File:lu67917r1ezu_tmp_b07ae624.png
</gallery>

<span id="configure-main-postfix-configuration---this-is-similar-to-what-we-did-for-freepbx-voicemail-alerts-in-the-previous-section"></span>
==== 1.2 Configure Main Postfix Configuration - this is similar to what we did for FreePBX voicemail alerts in the previous section ====

<ol style="list-style-type: decimal;">
<li><p>Backup existing configuration:</p>
<pre>sudo cp /etc/postfix/main.cf /etc/postfix/main.cf.backup</pre></li>
<li><p>Create new <code>main.cf</code>:</p>
<pre>sudo nano /etc/postfix/main.cf</pre></li>
<li><p>Copy and paste the provided configuration template if you need, and edit the <code>yourdriveisdead@stevesavers.com</code> email in the configuration file with the email you wish to have Postfix use to send you an email.</p></li></ol>

<pre>  # See /usr/share/postfix/main.cf.dist for a commented, more complete version
  
  
  # Debian specific:  Specifying a file name will cause the first
  # line of that file to be used as the name.  The Debian default
  # is /etc/mailname.
  #myorigin = /etc/mailname
  
  smtpd_banner = $myhostname ESMTP $mail_name (Debian/GNU)
  biff = no
  
  # appending .domain is the MUA's job.
  append_dot_mydomain = no
  
  # Uncomment the next line to generate &quot;delayed mail&quot; warnings
  #delay_warning_time = 4h
  
  readme_directory = no
  
  # See http://www.postfix.org/COMPATIBILITY_README.html -- default to 3.6 on
  # fresh installs.
  compatibility_level = 3.6
  
  
  
  # TLS parameters
  smtpd_tls_cert_file=/etc/ssl/certs/ssl-cert-snakeoil.pem
  smtpd_tls_key_file=/etc/ssl/private/ssl-cert-snakeoil.key
  smtpd_tls_security_level=may
  
  smtp_tls_CApath=/etc/ssl/certs
  smtp_tls_security_level=may
  smtp_tls_session_cache_database = btree:${data_directory}/smtp_scache
  
  
  smtpd_relay_restrictions = permit_mynetworks 
  permit_sasl_authenticated defer_unauth_destination
  myhostname = debian.home.arpa
  alias_maps = hash:/etc/aliases
  alias_database = hash:/etc/aliases
  mydestination = $myhostname, debian, localhost.localdomain, localhost
  relayhost = [smtp.postmarkapp.com]:587
  smtp_use_tls = yes
  smtp_sasl_auth_enable = yes
  smtp_sasl_password_maps = hash:/etc/postfix/sasl_passwd
  smtp_sasl_security_options = noanonymous
  smtp_sasl_mechanism_filter = plain
  sender_canonical_maps = static:yourdriveisdead@stevesavers.com
  mynetworks = 127.0.0.0/8 [::ffff:127.0.0.0]/104 [::1]/128
  mailbox_size_limit = 0
  recipient_delimiter = +
  # WARNING: Changing the inet_interfaces to an IP other than 127.0.0.1 may expose Postfix to external network connections.
  # Only modify this setting if you understand the implications and have specific network requirements.
  inet_interfaces = 127.0.0.1
  inet_protocols = all
  message_size_limit = 102400000</pre>
<span id="set-up-smtp-authentication-and-use-your-usernamespasswordsemails-to-replace-mine"></span>
==== 1.3 Set Up SMTP Authentication, and use your usernames/passwords/emails to replace mine ====

<ol style="list-style-type: decimal;">
<li><p>Create the SASL password file:</p>
<pre>sudo nano /etc/postfix/sasl_passwd</pre></li>
<li><p>Add this line to the file, replacing the username &amp; password with your credentials from postmark:</p></li></ol>

<pre>[smtp.postmarkapp.com]:587 1788dd83-9917-46e1-b90a-3b9a89c10bd7:1788dd83-9917-46e1-b90a-3b9a89c10bd7</pre>
<ol start="3" style="list-style-type: decimal;">
<li><p>Set proper permissions for security:</p>
<pre>sudo chmod 600 /etc/postfix/sasl_passwd</pre></li>
<li><p>Create the hash database file:</p>
<pre>sudo postmap /etc/postfix/sasl_passwd</pre></li></ol>

<span id="restart-and-test"></span>
==== 1.4 Restart and Test ====

<ol style="list-style-type: decimal;">
<li><p>Restart Postfix:</p>
<pre>sudo systemctl restart postfix</pre></li>
<li><p>Verify Postfix is running:</p>
<pre>sudo systemctl status postfix</pre></li>
<li><p>Test the email setup:</p>
<pre>echo &quot;Test email from $(hostname)&quot; | mail -s &quot;Test Email&quot; l.a.rossmann@gmail.com</pre></li></ol>

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:lu67917r1ezu_tmp_db123f98.png
File:lu67917r1ezu_tmp_cf91d8ae.png
</gallery> '''Verification Steps:'''

<ol style="list-style-type: decimal;">
<li><p>Check mail logs for errors:</p>
<pre>sudo tail -f /var/log/mail.log</pre></li>
<li><p>Verify permissions:</p>
<pre>ls -l /etc/postfix/sasl_passwd*</pre>
<p>Should show:</p>
<ul>
<li><code>-rw------- 1 root root</code> for sasl_passwd</li>
<li><code>-rw------- 1 root root</code> for sasl_passwd.db</li></ul>
</li></ol>

<span id="troubleshooting-1"></span>
=== Troubleshooting: ===

If emails aren’t being sent:

<ol style="list-style-type: decimal;">
<li><p>Check Postfix status:</p>
<pre>sudo systemctl status postfix</pre></li>
<li><p>View detailed mail logs:</p>
<pre>sudo journalctl -u postfix</pre></li></ol>

Check mail logs for errors:

<pre>sudo tail -f /var/log/mail.log</pre>
# Check <code>/var/log/mail.log</code> for errors
# Check that Postmark credentials are correct (e.g., if you typed <code>postmark.com</code> instead of <code>postmarkapp.com</code> for server, etc.)
# Verify sender domain (<code>stevesavers.com</code>) is properly configured in Postmark
# Check the '''Activity''' tab on the transactional stream in Postmark
# Mail log will tell you what you fkd up 99% of the time.

'''This setup does as follows:'''

* Send FROM: ''[mailto:yourdriveisdead@stevesavers.com yourdriveisdead@stevesavers.com]''
* Send TO: ''[mailto:l.a.rossmann@gmail.com l.a.rossmann@gmail.com]''
* Use the configured SMTP relay
* Include proper authentication

The system is now ready for the next step in the ZFS monitoring setup.

<span id="step-2-creating-complete-zfs-monitoring-script-with-logging"></span>
== Step 2: Creating Complete ZFS Monitoring Script with Logging ==

<span id="create-log-directory"></span>
==== 2.1 Create Log Directory ====

<pre>sudo mkdir -p /var/log/zfs-monitor
sudo chown root:root /var/log/zfs-monitor
sudo chmod 755 /var/log/zfs-monitor</pre>
<span id="make-the-monitoring-script"></span>
==== 2.2 Make the Monitoring Script ====

<pre>sudo -u root nano /root/zfs_health_check.sh</pre>
Copy and paste this complete script:

<pre>#!/bin/bash

# Configuration
EMAIL=&quot;l.a.rossmann@gmail.com&quot;
HOSTNAME=$(hostname)
LOG_FILE=&quot;/var/log/zfs-monitor/health_check.log&quot;
LOG_MAX_SIZE=$((10 * 1024 * 1024))  # 10MB in bytes

# Email configuration
FROM_EMAIL=&quot;yourdriveisdead@stevesavers.com&quot;
FROM_NAME=&quot;Steve&quot;
REPLY_TO=&quot;Steve &lt;steve@stevesavers.com&gt;&quot;  # Use a more consistent Reply-To address
RETURN_PATH=&quot;bounce@stevesavers.com&quot;  # A safe Return-Path address to handle bounces properly

# Create required directories
mkdir -p &quot;$(dirname &quot;$LOG_FILE&quot;)&quot;

# Initialize error log
errors=&quot;&quot;

# Logging functions
rotate_log() {
    if [ -f &quot;$LOG_FILE&quot; ] &amp;&amp; [ $(stat -f%z &quot;$LOG_FILE&quot; 2&gt;/dev/null || stat -c%s &quot;$LOG_FILE&quot;) -gt &quot;$LOG_MAX_SIZE&quot; ]; then
        mv &quot;$LOG_FILE&quot; &quot;$LOG_FILE.old&quot;
    fi
}

log_message() {
    echo -e &quot;[$(date '+%Y-%m-%d %H:%M:%S')] $1&quot; | tee -a &quot;$LOG_FILE&quot;
}

log_error() {
    local message=&quot;$1&quot;
    errors=&quot;${errors}\n$message&quot;
    log_message &quot;ERROR: $message&quot;
}

# Check overall pool status
check_pool_status() {
    while IFS= read -r pool; do
        status=$(zpool status &quot;$pool&quot;)
        
        # Check for common failure keywords
        if echo &quot;$status&quot; | grep -E &quot;DEGRADED|FAULTED|OFFLINE|UNAVAIL|REMOVED|FAIL|DESTROYED|SUSPENDED&quot; &gt; /dev/null; then
            log_error &quot;ALERT: Pool $pool is not healthy:\n$status&quot;
        fi
        
        # Check for errors
        if echo &quot;$status&quot; | grep -v &quot;No known data errors&quot; | grep -i &quot;errors:&quot; &gt; /dev/null; then
            log_error &quot;ALERT: Pool $pool has errors:\n$status&quot;
        fi
        
        # Check scrub status
        if echo &quot;$status&quot; | grep &quot;scan&quot; | grep -E &quot;scrub canceled|scrub failed&quot; &gt; /dev/null; then
            log_error &quot;ALERT: Pool $pool has unusual scrub status:\n$(echo &quot;$status&quot; | grep &quot;scan&quot;)&quot;
        fi
    done &lt; &lt;(zpool list -H -o name)
}

# Check individual device status
check_devices() {
    while IFS= read -r pool; do
        devices=$(zpool status &quot;$pool&quot; | awk '/ONLINE|DEGRADED|FAULTED|OFFLINE|UNAVAIL|REMOVED/ {print $1,$2}')
        
        echo &quot;$devices&quot; | while read -r device state; do
            if [ &quot;$state&quot; != &quot;ONLINE&quot; ] &amp;&amp; [ &quot;$device&quot; != &quot;pool&quot; ] &amp;&amp; [ &quot;$device&quot; != &quot;mirror&quot; ] &amp;&amp; [ &quot;$device&quot; != &quot;raidz1&quot; ] &amp;&amp; [ &quot;$device&quot; != &quot;raidz2&quot; ]; then
                log_error &quot;ALERT: Device $device in pool $pool is $state&quot;
            fi
        done
    done &lt; &lt;(zpool list -H -o name)
}

# Check capacity threshold (80% by default)
check_capacity() {
    while IFS= read -r pool; do
        capacity=$(zpool list -H -p -o capacity &quot;$pool&quot;)
        if [ &quot;$capacity&quot; -ge 80 ]; then
            log_error &quot;WARNING: Pool $pool is ${capacity}% full&quot;
        fi
    done &lt; &lt;(zpool list -H -o name)
}

# Check dataset properties
check_dataset_properties() {
    while IFS= read -r dataset; do
        # Skip base pools
        if ! echo &quot;$dataset&quot; | grep &quot;/&quot; &gt; /dev/null; then
            continue
        fi
        
        # Check if compression is enabled
        compression=$(zfs get -H compression &quot;$dataset&quot; | awk '{print $3}')
        if [ &quot;$compression&quot; = &quot;off&quot; ]; then
            log_error &quot;WARNING: Compression is disabled on dataset $dataset&quot;
        fi
        
        # Check if dataset is mounted
        mounted=$(zfs get -H mounted &quot;$dataset&quot; | awk '{print $3}')
        if [ &quot;$mounted&quot; = &quot;no&quot; ]; then
            log_error &quot;WARNING: Dataset $dataset is not mounted&quot;
        fi
        
        # Check available space
        available=$(zfs get -H available &quot;$dataset&quot; | awk '{print $3}')
        if [ &quot;$available&quot; = &quot;0&quot; ] || [ &quot;$available&quot; = &quot;0B&quot; ]; then
            log_error &quot;CRITICAL: Dataset $dataset has no available space&quot;
        fi
    done &lt; &lt;(zfs list -H -o name)
}

# Function to send email
send_email() {
    local subject=&quot;$1&quot;
    local content=&quot;$2&quot;
    
    {
        echo &quot;Subject: $subject&quot;
        echo &quot;To: ${EMAIL}&quot;
        echo &quot;From: ${FROM_NAME} &lt;${FROM_EMAIL}&gt;&quot;
        echo &quot;Reply-To: ${REPLY_TO}&quot;
        echo &quot;Return-Path: ${RETURN_PATH}&quot;
        echo &quot;Content-Type: text/plain; charset=UTF-8&quot;
        echo
        echo &quot;$content&quot;
    } | sendmail -t
}

# Main execution
rotate_log
log_message &quot;Starting ZFS health check&quot;

# Run all checks
check_pool_status
check_devices
check_capacity
check_dataset_properties

# Send notification if there are errors
if [ -n &quot;$errors&quot; ]; then
    log_message &quot;Issues detected - sending email alert&quot;
    subject=&quot;Storage Alert: Issues Detected on ${HOSTNAME}&quot;  # Simplified subject line
    content=$(echo -e &quot;ZFS Health Monitor Report from ${HOSTNAME}\n\nThe following issues were detected:${errors}&quot;)
    send_email &quot;$subject&quot; &quot;$content&quot;
else
    log_message &quot;All ZFS checks passed successfully&quot;
fi</pre>
<span id="set-proper-permissions-2"></span>

==== 2.3 Set Proper Permissions ====

<pre>sudo -u root chmod +x /root/zfs_health_check.sh</pre>
<span id="test-the-script"></span>
==== 2.4 Test the Script ====

<pre>sudo /root/zfs_health_check.sh</pre>
<span id="make-sure-logging-works"></span>
==== 2.5 Make sure logging works ====

<pre>tail -f /var/log/zfs-monitor/health_check.log</pre>
<span id="features-of-this-script"></span>
==== 2.6 Features of this Script: ====

* '''Monitoring''':
** It tells you when your pool has issues BEFORE all your drives die
** Device status checks
** Capacity warnings
* '''Email Alerts''':
** Sends when issues are detected
** Includes error information

The script is now ready for cron job configuration and regular use. Cron jobs are tasks we tell the machine to perform at regular intervals, similar to setting a utility bill to autopay.

<span id="step-3-create-cron-job"></span>
== Step 3: Create Cron Job ==

<ol style="list-style-type: decimal;">
<li><p>Open root’s crontab:</p>
<pre>sudo crontab -e</pre></li>
<li><p>Add these lines:</p>
<pre># ZFS Health Check - Run every 15 minutes
*/15 * * * * /root/zfs_health_check.sh &gt;/dev/null 2&gt;&amp;1

# Log rotation - Run daily at midnight
0 0 * * * find /var/log/zfs-monitor -name &quot;*.old&quot; -mtime +7 -delete</pre></li></ol>

<span id="step-4-verify-it-works-again-just-because"></span>
== Step 4: Verify it works again, just because ==

Run the script manually to ensure it works:

<pre>sudo /root/zfs_health_check.sh</pre>
<span id="check-logs"></span>
=== Check Logs ===

Monitor the log file for any issues:

<pre>tail -f /var/log/zfs-monitor/health_check.log</pre>
<span id="make-sure-cron-job-is-listed"></span>
=== Make sure Cron Job is listed ===

Verify that the cron job is correctly listed:

<pre>sudo crontab -l</pre>
<span id="test-email-notifications"></span>
=== Test Email Notifications ===

# Unplug a drive.
# Wait.
# Does an email come through?

The monitoring system is now fully configured and will:

* Check ZFS status every 15 minutes
* Log all checks to <code>/var/log/zfs-monitor/health_check.log</code>
* Automatically rotate logs when they reach 10MB
* Send email alerts only when issues are detected
* Clean up old log files after 7 days

<span id="how-to-tell-if-you-won"></span>
== How to tell if you won: ==

* ✓ Test email received
* ✓ Script detects simulated issues
* ✓ Cron job executes on schedule
* ✓ Logs show proper entries
* ✓ Alerts generated for pool degradation
* ✓ System returns to normal after tests

If you got an email, congrats, it works!

<span id="step-5-set-up-os-raid-array-to-email-you-when-theres-a-problem-as-well"></span>
== Step 5: Set up OS RAID Array to email you when there’s a problem as well ==

What we set up above is for your '''''ARCHIVE''''' storage. What about your operating system? We will do the same thing, and also go over a barbaric backup routine that works for me.

<span id="creating-the-alert-script"></span>
==== 5.1 Creating the alert script ====

I’m not a programmer, so bear with me. This script is for my personal use, but I’m sharing it because it works. Here’s what you need to do:

# '''Edit Email Addresses''': You’ll need to change the email addresses in the script. This includes:
#* The recipient email
#* The sender email
#* The reply-to address
#* The return path for bounced emails
# '''Script Location''': Save the script at <code>root/mdadm_alert.sh</code>

<pre>sudo -u root nano -w /root/mdadm_alert.sh</pre>
Enter the following:

<pre>#!/bin/bash

# thank you to stack overflow for giving me the courage to wade through 100s of posts and hack together something that looks like it works. 

# stricter error handling
set -euo pipefail  # ‘set -e’ exits on errors, ‘u’ throws errors on unset variables, &amp; ‘pipefail’ exits if any part of a pipeline fails
IFS=$'\n\t'  # Set IFS (Internal Field Separator) to newline &amp; tab to avoid issues with spaces and other weird characters in filenames

# Configuration variables (where settings are stored)
EMAIL=&quot;l.a.rossmann@gmail.com&quot;  # Email to send alerts to - EDIT THIS 
HOSTNAME=$(hostname)  # Pull the system's hostname dynamically and save it here
LOG_DIR=&quot;/var/log/mdadm-monitor&quot;  # Directory path for where logs go
LOG_FILE=&quot;${LOG_DIR}/raid_health_check.log&quot;  # Full path to the specific log file for RAID checks
LOG_MAX_SIZE=$((10 * 1024 * 1024))  # Maximum log file size in bytes (10 MB here)

# Email configuration for the alert message
FROM_EMAIL=&quot;yourdriveisdead@stevesavers.com&quot;  # The email address that will appear as the sender - EDIT THIS
FROM_NAME=&quot;Steve&quot;  # name of the sender, EDIT THIS
REPLY_TO=&quot;Steve &lt;steve@stevesavers.com&gt;&quot;  # Reply-to email address, EDIT THIS
RETURN_PATH=&quot;bounce@stevesavers.com&quot;  # Return path for bounced emails when email fails EDIT THIS

# make empty variables &amp; associated arrays 
errors=&quot;&quot;  # Empty variable to collect error messages
drive_health_report=&quot;&quot;  # Another empty variable to store drive health details
declare -A RAID_ARRAYS  # array to keep track of RAID arrays we find, indexed by  name like &quot;boot&quot;
declare -A SMART_SCORES  # array to store SMART scores for drives, indexed by rive path

# Set up log directory and ensure permissions are correct
setup_logging() {
    # Make the log directory if it doesn’t already exist
    mkdir -p &quot;$LOG_DIR&quot; || { echo &quot;ERROR: Cannot create log directory $LOG_DIR&quot;; exit 1; }  # Exit with error if I can’t make the directory
    chmod 750 &quot;$LOG_DIR&quot;  # Set directory permissions to allow owner &amp; group access but not others

    # Check if the log file exists and exceeds the max size limit
    if [ -f &quot;$LOG_FILE&quot; ] &amp;&amp; [ &quot;$(stat -c%s &quot;$LOG_FILE&quot;)&quot; -gt &quot;$LOG_MAX_SIZE&quot; ]; then  # ‘stat -c%s’ gives the size in bytes
        mv &quot;$LOG_FILE&quot; &quot;$LOG_FILE.old&quot;  # Archive the old log file by renaming it
    fi
    touch &quot;$LOG_FILE&quot;  # Create an empty log file if it doesn’t exist
    chmod 640 &quot;$LOG_FILE&quot;  # Set permissions on the log file (read/write for owner, read for group)
}

# Function for logging messages w/ timestamps
log_message() {
    local timestamp  # Make local variable for this
    timestamp=$(date '+%Y-%m-%d %H:%M:%S')  # Generate a timestamp in this specific format
    echo &quot;[$timestamp] $1&quot; | tee -a &quot;$LOG_FILE&quot;  # Output the message with the timestamp to both console &amp; log file
}

# Function for logging errors (adds them to the error string and logs them as &quot;ERROR&quot;)
log_error() {
    local message=&quot;$1&quot;  # Message passed to this function
    errors=&quot;${errors}\n$message&quot;  # Append this message to the errors variable
    log_message &quot;ERROR: $message&quot;  # Log the error with a timestamp
}

# Check that required (commands) are installed on the system
check_dependencies() {
    log_message &quot;Checking required dependencies...&quot;  # Announce the check in the log
    local missing_deps=()  # Initialize an empty array for any missing commands

    # Loop through each command we need, checking if it’s available
    for dep in mdadm smartctl lsblk findmnt awk grep dmsetup; do
        if ! command -v &quot;$dep&quot; &amp;&gt;/dev/null; then  # If the command is missing, add it to the array
            missing_deps+=(&quot;$dep&quot;)
        fi
    done

    # If the array of missing dependencies isn’t empty, log an error and exit
    if [ ${#missing_deps[@]} -ne 0 ]; then
        log_error &quot;Missing required dependencies: ${missing_deps[*]}&quot;  # Log missing commands
        log_error &quot;Install them with: sudo apt-get install mdadm smartmontools util-linux findutils gawk grep dmsetup&quot;
        exit 1  # Exit with error because we’re missing something we need(find what you need if you're getting this)
    fi
}

# Find &amp; detect RAID arrays on this system
detect_raid_arrays() {
    log_message &quot;Detecting RAID arrays...&quot;  # Log that we’re looking for RAID arrays

    # Find all block devices with names like /dev/md0, /dev/md1 (these are RAID arrays like the one you made for the OS &amp; boot)
    local md_devices
    md_devices=$(find /dev -name 'md[0-9]*' -type b)  # Save this list to the md_devices variable

    # Loop through each RAID array found and log its details
    for md_dev in $md_devices; do
        local array_detail  # Temporary variable for array details
        array_detail=$(mdadm --detail &quot;$md_dev&quot; 2&gt;/dev/null) || continue  # Get RAID details; skip if it fails

        # Extract the RAID array name from the details
        local array_name
        array_name=$(echo &quot;$array_detail&quot; | grep &quot;Name&quot; | awk '{print $NF}')  # Last word on the &quot;Name&quot; line is the array name

        # Use the name to decide if this array is for boot or root, then add it to RAID_ARRAYS
        if [[ &quot;$array_name&quot; == *&quot;bootraid&quot;* ]]; then  # Array name contains &quot;bootraid&quot;
            RAID_ARRAYS[&quot;boot&quot;]=&quot;$md_dev&quot;  # Save the device path with the key &quot;boot&quot;
            log_message &quot;Found boot array: $md_dev ($array_name)&quot;  # Log the found boot array
        elif [[ &quot;$array_name&quot; == *&quot;osdriveraid&quot;* ]]; then  # Array name contains &quot;osdriveraid&quot;
            RAID_ARRAYS[&quot;root&quot;]=&quot;$md_dev&quot;  # Save the device path with the key &quot;root&quot;
            log_message &quot;Found root array: $md_dev ($array_name)&quot;  # Log the found root array
        fi
    done

    # Check if we actually found both root and boot arrays, and log an error if any are missing
    if [ -z &quot;${RAID_ARRAYS[&quot;boot&quot;]:-}&quot; ] || [ -z &quot;${RAID_ARRAYS[&quot;root&quot;]:-}&quot; ]; then  # If either key is empty
        log_error &quot;Failed to detect both boot and root RAID arrays&quot;  # Log a general error
        [ -z &quot;${RAID_ARRAYS[&quot;boot&quot;]:-}&quot; ] &amp;&amp; log_error &quot;Boot array not found&quot;  # Specific message if boot is missing
        [ -z &quot;${RAID_ARRAYS[&quot;root&quot;]:-}&quot; ] &amp;&amp; log_error &quot;Root array not found&quot;  # Specific message if root is missing
        return 1  # Return an error code
    fi

    # Print out a summary of all arrays found
    log_message &quot;Detected arrays:&quot;
    for purpose in &quot;${!RAID_ARRAYS[@]}&quot;; do
        log_message &quot;  $purpose: ${RAID_ARRAYS[$purpose]}&quot;
    done
}

# Check the health of a specific RAID array
check_array_status() {
    local array=&quot;$1&quot;  # The path of the array device
    local purpose=&quot;$2&quot;  # Either &quot;boot&quot; or &quot;root&quot; to clarify which array this is

    # Verify that the array actually exists as a block device
    if [ ! -b &quot;$array&quot; ]; then
        log_error &quot;$purpose array device $array does not exist&quot;  # Log the missing device
        return 1  # Return error because we can’t check a nonexistent device
    fi

    # Get details about the RAID array and store it in the detail variable
    local detail
    detail=$(mdadm --detail &quot;$array&quot; 2&gt;&amp;1) || {  # ‘2&gt;&amp;1’ captures error output in case of issues
        log_error &quot;Failed to get details for $purpose array ($array)&quot;
        return 1  # Exit with an error code if it failed
    }

    # Extract the state of the array (like &quot;clean&quot; or &quot;active&quot;) and log it
    local state
    state=$(echo &quot;$detail&quot; | grep &quot;State :&quot; | awk '{print $3,$4}')  # Get the words after &quot;State :&quot; from the details
    log_message &quot;$purpose array status: $state&quot;

    # If the array is in an undesirable state, log a warning
if [[ &quot;$state&quot; =~ degraded|DEGRADED|failed|FAILED|inactive|INACTIVE ]]; then
    log_error &quot;$purpose array ($array) is in concerning state: $state&quot;
fi

    # Detect failed devices within the array
    local failed_devices
    failed_devices=$(echo &quot;$detail&quot; | grep &quot;Failed Devices&quot; | awk '{print $4}')  # Pull the failed devices count
    if [ &quot;$failed_devices&quot; -gt 0 ]; then  # If there are failed devices, go through each one
        while read -r line; do
            if [[ &quot;$line&quot; =~ &quot;faulty&quot; ]]; then  # If the line mentions &quot;faulty&quot;
                local failed_dev
                failed_dev=$(echo &quot;$line&quot; | awk '{print $7}')  # Get the 7th word (the device name)
                log_error &quot;$purpose array ($array) has failed device: $failed_dev&quot;  # Log which device failed
            fi
        done &lt; &lt;(echo &quot;$detail&quot; | grep -A20 &quot;Number&quot; | grep &quot;faulty&quot;)  # Look up to 20 lines after &quot;Number&quot; to find &quot;faulty&quot;
    fi

    # Check if any devices are rebuilding, and log it if they are
    if echo &quot;$detail&quot; | grep -q &quot;rebuilding&quot;; then
        while read -r line; do
            if [[ &quot;$line&quot; =~ &quot;rebuilding&quot; ]]; then  # Check for &quot;rebuilding&quot; in the line
                local rebuilding_dev
                rebuilding_dev=$(echo &quot;$line&quot; | awk '{print $7}')  # Get the device name being rebuilt
                log_error &quot;$purpose array ($array) is rebuilding device: $rebuilding_dev&quot;  # Log the rebuilding device
            fi
        done &lt; &lt;(echo &quot;$detail&quot; | grep -A20 &quot;Number&quot; | grep &quot;rebuilding&quot;)  # Again, look ahead 20 lines for any &quot;rebuilding&quot; mention
    fi
}

# Function to check the health of each drive within a RAID array
check_drive_health() {
    local drive=&quot;$1&quot;  # The drive device to check (e.g., /dev/sda)
    local health_score=100  # Initialize health score to 100 (a perfect score)
    local issues=&quot;&quot;

    # Skip the check if it’s not a valid block device
    if [ ! -b &quot;$drive&quot; ]; then
        log_error &quot;Device $drive is not a block device&quot;  # Log the invalid device
        return 1  # Exit with an error code
    fi

    log_message &quot;Checking health of drive $drive...&quot;  # Announce which drive we’re checking

    # Run SMART health check and reduce health score if it fails
    if ! smartctl -H &quot;$drive&quot; | grep -q &quot;PASSED&quot;; then  # If it does NOT say &quot;PASSED&quot;
        health_score=$((health_score - 50))  # Drop score by 50 points if it fails
        issues+=&quot;\n- Overall health check failed&quot;  # Log this specific issue
    fi

    # Collect SMART attributes for further checks
    local smart_attrs
    smart_attrs=$(smartctl -A &quot;$drive&quot; 2&gt;/dev/null) || true  # Redirect error to /dev/null

    # Check for reallocated sectors (sign of drive wear and tear)
    local reallocated
    reallocated=$(echo &quot;$smart_attrs&quot; | awk '/^  5/ {print $10}')  # Look for attribute ID 5 in SMART data
    if [ -n &quot;$reallocated&quot; ] &amp;&amp; [ &quot;$reallocated&quot; -gt 0 ]; then
        health_score=$((health_score - 10))  # Drop health score by 10 if we have reallocated sectors
        issues+=&quot;\n- Reallocated sectors: $reallocated&quot;  # Add to issues list
    fi

    # Check for pending sectors (could cause read/write errors)
    local pending
    pending=$(echo &quot;$smart_attrs&quot; | awk '/^197/ {print $10}')  # Look for attribute ID 197 in SMART data
    if [ -n &quot;$pending&quot; ] &amp;&amp; [ &quot;$pending&quot; -gt 0 ]; then
        health_score=$((health_score - 10))  # Drop health score by 10 if pending sectors are present
        issues+=&quot;\n- Pending sectors: $pending&quot;  # Add to issues list
    fi

    SMART_SCORES[&quot;$drive&quot;]=$health_score  # Save the final score in SMART_SCORES array
    if [ &quot;$health_score&quot; -lt 100 ]; then
        drive_health_report+=&quot;\nDrive: $drive\nHealth Score: $health_score/100\nIssues:$issues&quot;  # Append issues to report if any were found
    fi
}

# Send email if any errors or health issues were found
send_email() {
    local subject=&quot;RAID Alert: Issues Detected on ${HOSTNAME}&quot;  # Set email subject line
    local content=&quot;RAID Health Monitor Report from ${HOSTNAME}\nTime: $(date '+%Y-%m-%d %H:%M:%S')\n&quot;
    [ -n &quot;$errors&quot; ] &amp;&amp; content+=&quot;\nRAID Issues:${errors}&quot;  # Append RAID issues to the email content if any
    [ -n &quot;$drive_health_report&quot; ] &amp;&amp; content+=&quot;\nDrive Health Report:${drive_health_report}&quot;  # Append drive health report if any issues were found

    # Build the email using sendmail syntax
    {
        echo &quot;Subject: $subject&quot;
        echo &quot;To: ${EMAIL}&quot;
        echo &quot;From: ${FROM_NAME} &lt;${FROM_EMAIL}&gt;&quot;
        echo &quot;Reply-To: ${REPLY_TO}&quot;
        echo &quot;Return-Path: ${RETURN_PATH}&quot;
        echo &quot;Content-Type: text/plain; charset=UTF-8&quot;  # Text format for readability
        echo
        echo -e &quot;$content&quot;  # Use ‘-e’ to allow newline characters
    } | sendmail -t  # Pipe the entire email message to sendmail for delivery
}

# Main function to execute checks and send email if needed
main() {
    # Make sure script is run as root for necessary permissions
    [ &quot;$(id -u)&quot; -ne 0 ] &amp;&amp; { echo &quot;ERROR: This script must be run as root&quot;; exit 1; }
    setup_logging  # Call function to initialize logging setup
    log_message &quot;Starting RAID health check&quot;  # Announce the start of the health check
    check_dependencies  # Verify dependencies are available
    detect_raid_arrays  # Detect RAID arrays

    # Loop through each RAID array and check its status, then check each drive in the array
    for purpose in &quot;${!RAID_ARRAYS[@]}&quot;; do
        array=&quot;${RAID_ARRAYS[$purpose]}&quot;
        check_array_status &quot;$array&quot; &quot;$purpose&quot;

        # For each device in the RAID array, check health
        while read -r device; do
            if [[ &quot;$device&quot; =~ ^/dev/ ]]; then
                check_drive_health &quot;$device&quot;
            fi
        done &lt; &lt;(mdadm --detail &quot;$array&quot; | grep &quot;active sync&quot; | awk '{print $NF}')
    done

    # Send an email if errors or health issues were found; otherwise, log a success message
    [ -n &quot;$errors&quot; ] || [ -n &quot;$drive_health_report&quot; ] &amp;&amp; send_email || log_message &quot;All checks passed successfully&quot;
}

# Execute the main function to start everything
main  # Calls the main function, running all the checks</pre>
Set permissions properly so it can run:

<pre>sudo -u root chmod +x /root/mdadm_alert.sh</pre>
<span id="setting-up-the-cron-job"></span>

==== 5.2 Setting Up the Cron Job ====

We want this script to run regularly. I am going to set it to run every 15 minutes.

<pre># Open the crontab editor
sudo -u root crontab -e</pre>
Add the following line to run the script every minute (for testing purposes):

<pre>* * * * * /root/mdadm_alert.sh</pre>
<blockquote>'''Note:''' For regular use, set it to run every fifteen minutes, with a line such as <code>*/15 * * * * /root/mdadm_alert.sh</code>
</blockquote>
<span id="testing-the-setup---software-run-first."></span>
==== 5.3 Testing the setup - software run first. ====

Let’s simulate a fault condition on <code>/dev/md126</code> which is what I set up as the RAID1 array for the operating system installation; this is where we created the logical volume for <code>/</code>

# Check the status of it as it is now:

<pre>sudo mdadm --detail /dev/md126</pre>
<ol start="2" style="list-style-type: decimal;">
<li>If it shows up as healthy, run the script to make sure we do not have false positives.</li></ol>

<pre>sudo -u root /root/mdadm_alert.sh</pre>
<ol start="3" style="list-style-type: decimal;">
<li>If no false positives, simulate fault condition:</li></ol>

<pre>sudo mdadm /dev/md126 --fail /dev/sdb3</pre>
<code>/dev/sdb3</code> was the drive &amp; partition that was used in my RAID array. Yours may differ, refer to the output of <code>mdadm --detail</code> to see how your RAID array is comprised, and then fail one of the two devices.

<ol start="4" style="list-style-type: decimal;">
<li>Run the monitoring script to test again.</li></ol>

<pre>sudo -u root /root/mdadm_alert.sh</pre>
You should receive an email. Check spam.

<ol start="5" style="list-style-type: decimal;">
<li>Undo what you did, un-fail the drive.</li></ol>

<pre>sudo mdadm /dev/md126 --remove /dev/sdb3
sudo mdadm /dev/md126 --add /dev/sdb3</pre>
<ol start="6" style="list-style-type: decimal;">
<li>Watch it re-sync. Don’t mess with anything until it is fully resynced.</li></ol>

<pre>watch cat /proc/mdstat</pre>
<span id="testing-the-setup-for-real---hardware-fault."></span>
==== 5.4 Testing the setup for real - hardware fault. ====

Now, let’s test this setup. Unplug one of the drives and see if you get a failure alert. Obviously, don’t do this after you start storing anything important on here. We do this in the build phase of our system to make sure it works, BEFORE trusting this system with anything important.

# Check the status of it as it is now:

<pre>sudo mdadm --detail /dev/md126</pre>
<ol start="2" style="list-style-type: decimal;">
<li>If it shows up as healthy, run the script to make sure we do not have false positives.</li></ol>

<pre>sudo -u root /root/mdadm_alert.sh</pre>
<ol start="3" style="list-style-type: decimal;">
<li>If no false positives, unplug the drive from the running system.</li></ol>

<code>/dev/sdb3</code> was the drive &amp; partition that was used in my RAID array. Yours may differ, refer to the output of <code>mdadm --detail</code> to see how your RAID array is comprised, and then fail one of the two devices.

<ol start="4" style="list-style-type: decimal;">
<li>Run the monitoring script to test again.</li></ol>

<pre>sudo -u root /root/mdadm_alert.sh</pre>
You should receive an email. Check spam.

<ol start="5" style="list-style-type: decimal;">
<li>Undo what you did, un-fail the drive after plugging it back in..</li></ol>

<pre>sudo mdadm /dev/md126 --remove /dev/sdb3
sudo mdadm /dev/md126 --add /dev/sdb3</pre>
<ol start="6" style="list-style-type: decimal;">
<li>Watch it re-sync. Don’t mess with anything until it is fully resynced.</li></ol>

<pre>watch cat /proc/mdstat</pre>
<span id="step-6-backup-strategy"></span>

== Step 6: Backup Strategy ==

Now, let’s talk about backups. It’s not enough to just have a RAID setup; you need a backup plan for when carelessness strikes.

<span id="backup-method"></span>
==== 6.1 Backup Method ====

Here’s my approach:

* '''Physical Copy''': I make a physical copy of my disk. This might seem old-school, but it works for me.

Another approach:

* '''LVM Snapshots''': You can take an LVM snapshot and then use <code>rsync</code> to back up your data. This method can be hit or miss. I don’t use this.

You can take a snapshot of your drive with LVM, rsync your files off of the drive elsewhere, reinstall the operating system, and rsync them back, but… what if some of your files are for older libraries, or programs/configuration files that have different syntax with different versions? It can become a rabbit hole to hell very easily, and I’m not going to begin to torture newbies with this.

'''DDRescue''' is the tool I use to make a copy of my drive. I connect the drive via a USB 3 to SATA plug and create a backup. It’s best to do this to the same make/model of drive if possible.

<span id="ddrescue-guide-from-ubuntu-server-live-environment"></span>

==== 6.2 DDRescue Guide from Ubuntu Server Live Environment ====

We’re going to boot from the same Ubuntu Server LiveUSB image you created to install Ubuntu Server onto the happycloud host machine.

* Boot from the USB Drive

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:lu55028jxc7f_tmp_911d702.png
File:lu55028jxc7f_tmp_a33d9a7f.png
</gallery>

# Insert the USB drive into your server.
# Power on the server and enter the boot menu (usually by pressing '''F12''' or another function key).
# Select the '''UEFI option''' for your USB drive.
# Choose to Try Ubuntu Server &amp; do not install it.

* Install ddrescue

# Update package list &amp; install ddrescue:

<pre>sudo apt update
sudo add-apt-repository universe
sudo apt install gddrescue</pre>
<ol start="2" style="list-style-type: decimal;">
<li>Check Current Drives (BEFORE Plugging in Source)</li></ol>

<pre>sudo fdisk -l</pre>
Take note of the present drives.

<ol start="3" style="list-style-type: decimal;">
<li><p>Connect Source Drive (operating system solid state drive from the happycloud host machine). Either will do. Either connect it physically to an existing SATA/NVME port, or use a USB-SATA or USB-NVME enclosure if this makes it easier for you.</p></li>
<li><p>Wait 5-10 seconds. Be patient.</p></li>
<li><p>Check which drive it is. It will be the new drive that shows up. Make sure the model as well as the size &amp; partitions matches what you are expecting.</p>
<pre>sudo fdisk -l</pre></li>
<li><p>Connect Target Drive (blank identical disk you are making into a backup drive)</p></li>
<li><p>Wait 5-10 seconds. Be patient.</p></li>
<li><p>Check which drive it is. It will be the new drive that shows up. Make sure the model as well as the size &amp; partitions matches what you are expecting.</p>
<pre>sudo fdisk -l</pre></li></ol>

'''TRIPLE CHECK YOUR DEVICES'''

<pre># List all drives again
sudo fdisk -l</pre>
<ol start="9" style="list-style-type: decimal;">
<li>Run DDRescue</li></ol>

<pre>sudo ddrescue -f -d -r3 /dev/source /dev/target logfile.log</pre>
For instance, if the source is <code>/dev/sdc</code> &amp; target is <code>/dev/sdd</code>:

<pre>sudo ddrescue -f -d -r3 /dev/sdc /dev/sdd logfile.log</pre>
Option meanings: - <code>-f</code> : Force overwrite target - <code>-d</code> : Use direct disk access - <code>-r3</code>: Number of retry attempts on bad sectors - logfile.log: Saves progress (can resume if interrupted)

'''⚠️ WARNING: ⚠️'''

<ol style="list-style-type: decimal;">
<li><p>TRIPLE CHECK device names</p>
<ul>
<li>Wrong device = destroyed data</li>
<li>Source and target reversed = destroyed source</li></ul>
</li>
<li><p>Target MUST be same size or larger than source</p></li>
<li><p>Make sure you’re using whole drives:</p>
<ul>
<li><code>/dev/sdc</code> (correct, whole drive)</li>
<li><code>/dev/sdc1</code> (WRONG, just one partition)</li></ul>
</li>
<li><p>If unsure which is which, unplug/replug and watch:</p>
<pre>sudo dmesg | tail</pre>
<p>It will show new devices added to the linux machine</p></li></ol>

<blockquote>'''IMPORTANT NOTE:''' Always have a physical copy of a known-working server solid state drive. If something wrong, you can quickly restore your system by plugging in the backup drive and be back up in 90 seconds or less.
</blockquote>
<span id="raid-configuration-recommendations"></span>
== RAID Configuration Recommendations ==

<ul>
<li><p>For those who are extra cautious, consider running a RAID 1 setup with '''three''' drives instead of '''two'''. Here’s why:</p>
<ul>
<li>'''Redundancy''': When one drive fails, the others are likely not far behind. Having a third drive adds some padding.</li>
<li>'''Peace of mind''': If you’re paranoid about data loss, this setup is a safer bet.</li></ul>

<p>If you wanted to avoid stressing the SSD, you could create a ZFS dataset on the ZFS pool of hard drives you set up for virtual machines, mount that as <code>/var/lib/libvirt/images/</code>, but I’ve gotten spoiled by the speed of SSDs - I don’t want to go back. I realize that writing to them a lot means killing them, and I’m ok with that. :)</p></li></ul>

<span id="os-drive-backup-conclusion"></span>
== OS drive backup conclusion: ==

Once everything is set up the way you like, shut down your system, remove one of the drives, and make a backup. Use a drive of equal or greater size for the backup. This way, if disaster strikes, you can restore your system in no time.

We now have a simple &amp; effective way to know when our operating system drive is about to die on us, so we can take action before anything horrible occurs. Best of all, if you set this up properly, you can have zero downtime &amp; not even have to turn off the machine to get back up and running when a drive fails.

<span id="setting-up-immich-google-photosicloud-replacement"></span>
= Setting Up Immich: Google Photos/iCloud replacement =

<span id="what-is-immich"></span>
== What is immich? ==

Immich is like Google photos or iCloud, if you hosted it yourself; but better! It has the following features that make it stand out to me:

<span id="why-immich"></span>
== Why Immich? ==

<span id="insanely-fast"></span>
==== Insanely fast ====

'''Immich''' loads &amp; scrolls through things on a core i3 NUC with an old SATA drive faster than '''nextcloud''' allowed me to on an i7-14700k with an NVME SSD. it’s snappy even on slower computers &amp; phones.

'''Nextcloud''' made the experience of browsing through images &amp; photos not on my phone so bad I stopped doing it; a flagship phone and an i7-14700k, 64 GB of RAM, and a $400 SSD wasn’t good enough to make this usable.

<span id="machine-learning-for-image-search"></span>
==== Machine learning for image search ====

I can type ''“cat on chair”'' and have every image of a cat on a chair show up. It actually works, it isn’t half assed and full of false positives.

Immich’s machine learning features &amp; included libraries are also used for '''face detection'''. Immich can sort your images by people, so you can see every image with your dad, cousin, girlfriend, ex-girlfriend, etc.

You can [https://huggingface.co/immich-app choose the model you want to use]. The default model works best for me, but I appreciate Immich respecting my right to choose the model I want.

Immich’s machine learning is done LOCALLY. '''Immich can be blocked from connecting to the internet and all machine learning &amp; facial recognition will still work.'''

When people hear the words ''“Artificial intelligence,”'' ''“cloud,”'' &amp; ''“machine learning,”'' they hear buzzwords for processes which were supposed to be used for our benefit, but instead have become tools of data mining &amp; abusive models. These are not bad things when they are done in a freedom-respecting way. I have no problem with machine learning algorithms going through all of my photos &amp; videos &amp; knowing the names of the people in my photos, because that information will never leave my computer.

<span id="easy-proxies"></span>

==== Easy proxies ====

Immich supports video &amp; image proxy files. Proxies are photos &amp; videos that are further compressed. They are lower in quality, but their smaller size allows you to load them faster when you’re on a poor internet connection. I use a google pixel, so this is handy. Google pixels have horrible cellphone service &amp; reception because Google is too stubborn to use Qualcomm modems. Google decided that its users care more about [https://www.reddit.com/r/GooglePixel/comments/1etk3l6/nothing_makes_me_less_excited_about_pixel_than/ lame AI features] than [https://www.reddit.com/r/GooglePixel/comments/x61kee/today_i_have_measured_just_how_bad_my_pixel_6s/ having working cell service]; This is where image proxies &amp; video proxies come in handy.

'''Nextcloud''' allows image proxies(with config file editing; ew). Immich allows both ''image'' AND ''video'' proxies, so high bitrate videos can still be loaded &amp; viewed on slow internet.

<span id="ease-of-use"></span>
==== Ease of use ====

This program is so easy to use you’ll almost forget you’re using GNU/Linux. When I set up my Nextcloud instance, I had to edit config files to get thumbnails to work. Further, nextcloud only allows ''image'' thumbnails, but not ''video'' proxies. Not only is it more work with nextcloud to get thumbnails &amp; proxies so you have something that loads well on a slow connection - it’s not as functional. Everything here is doable within the web interface after installation, and it’s easy as can be.

This program has the easiest installation &amp; documentation I’ve found for this type of GNU/Linux software. It is useless for me to provide instructions here because [following Immich team’s instructions, this will all work perfectly(https://immich.app/docs/install/docker-compose/) with no confusion. Immich is as good as as bitwarden with regards to “just working” out of the box &amp; a big part of why I fell in love with their progam.

<span id="prerequisites-1"></span>
== Prerequisites ==

Before starting, ensure you have:

* Docker Compose version 2.x installed(you should’ve done this setting up onlyoffice on this VM earlier)
* Docker installed from the official Docker repository(you should’ve done this setting up onlyoffice on this VM earlier)
* Enough storage space for the photos &amp; videos from your phone
* Did I mention not to install docker using SNAP from the ubuntu install? Don’t do that.

<span id="step-1-install-docker-properly."></span>
=== Step 1: Install docker properly. ===

<blockquote>'''NOTE:''' This step may not be necessary!
</blockquote>
**YOU DO NOT NEED TO PERFORM THIS STEP IF YOU INSTALLED DOCKER WHILE INSTALLING ONLYOFFICE. IF YOU INSTALLED DOCKER PRIOR TO INSTALLING ONLYOFFICE, SKIP THIS STEP! IF YOU DID NOT INSTALL ONLYOFFICE BECAUSE YOU DIDN’T WANT ONLYOFFICE, THAT MEANS YOU SKIPPED INSTALLING DOCKER AS WELL; IN WHICH CASE, YOU WILL NEED TO FOLLOW THESE INSTRUCTIONS.

<span id="never-use-ubuntus-snap-version-of-docker-1"></span>
==== Never use Ubuntu’s snap version of docker ====

Ubuntu installs docker by default using the cancerous snap. We do not want to use snap. Ubuntu installer will ask if you want to install Docker, and you should always say No. 

<span id="doesnt-onlyoffices-install-script-install-docker-for-me-1"></span>
==== Doesn’t onlyoffice’s install script install docker for me? ====

Onlyoffice’s installation script '''DOES''' install docker for you. I am still going to have you do it manually.

* If you choose to not install onlyoffice, and wish to install Immich, I want you to know how to install docker on this virtual machine ''yourself.''
* I don’t want to rely on onlyoffice’s script. It won’t install docker for us if it detects Docker already, so we’re not going to do a double install. What if onlyoffice’s installation script stops installing docker the same way in a new version, or stops installing docker at all within its script?

It’s little work to install Docker the right way for our purposes manually, and it’s good to have it documented so that you can use docker for immich even if you elect not to install Onlyoffice.

<span id="update-and-upgrade-your-system-3"></span>
==== 1.1 Update and upgrade your system ====

<pre>sudo apt update &amp;&amp; sudo apt upgrade -y
sudo apt install curl git wget -y</pre>
<span id="check-for-other-docker-installations-2"></span>
==== 1.2 Check for other Docker installations: ====

Run <code>docker --version</code> and see what is installed. Nothing should be installed yet since this is a fresh system. If something is installed, remove it.

<pre># Just in case you accidentally installed snap version of docker:

sudo snap remove docker

# For other versions of docker: 

sudo apt remove docker docker-engine docker.io containerd runc</pre>
<span id="install-docker-using-official-docker-script-3"></span>
==== 1.3 Install Docker using official Docker script: ====

<pre>curl -fsSL https://get.docker.com -o get-docker.sh
sudo sh get-docker.sh</pre>
<blockquote>'''Note:''' It’s very important to use the official Docker installation and not the Snap version. The Snap version can cause issues due to its sandboxed nature, making it a mess for mailcow’s requirements. Docker snap makes me sad, and it’ll make you sad too if you try to make things work with it.
</blockquote>
<span id="install-docker-compose-2"></span>
==== 1.4 Install Docker Compose: ====

Ubuntu’s <code>docker-compose-plugin</code> is safe to use, it is not snap cancer.

<pre>sudo apt install docker-compose-plugin -y
sudo systemctl enable --now docker</pre>
<span id="verify-the-install-2"></span>
==== 1.5 Verify the install ====

Run <code>docker compose version</code> and make sure the version is 2.0 or higher. Run <code>docker --version</code> and make sure version is 24.0.0 or higher

<span id="set-proper-permissions-3"></span>
==== 1.6 Set proper permissions: ====

Docker needs to be run as root for some operations, but you can add your user to the docker group to avoid using <code>sudo</code> all the time. To be clear, mailcow’s own [https://docs.mailcow.email/getstarted/install/#check-selinux-specifics documentation] and [https://community.mailcow.email/d/59-mailcow-containers-running-as-root community] suggest starting with root or <code>sudo</code>, and you should trust them more than me. To quote mailcow developers, ''“Controlling the Docker daemon as non-root user does not give you additional security. The unprivileged user will spawn the containers as root likewise. The behaviour of the stack is identical.”'' Run this command to add your user:

<pre>sudo usermod -aG docker $USER</pre>
Log out and log back in, or run:
<code>newgrp docker</code>

<div class="figure">

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:immichdiagrambad.png
</gallery>

</div>
<div class="figure">

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:immichok.png
</gallery>

</div>
<div class="figure">

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:immichdiagramgood.png
</gallery>

</div>
<span id="step-2-understand-how-this-will-be-set-up-differently-from-stock-setup."></span>
== Step 2: Understand how this will be set up differently from stock setup. ==

<span id="how-youre-supposed-to-use-immich"></span>
==== 2.1 How you’re supposed to use Immich ====

The stock setup of Immich, by default, is to have Immich upload your images &amp; videos from your phone to the immich server. You control your library on your phone &amp; on your server in the immich application.

<span id="syncthing-conflict-with-immich"></span>
==== 2.2 Syncthing conflict with Immich ====

Didn’t we already set up syncthing to do this? Yes, we did!

I don’t want to use Immich to sync my phone’s '''DCIM/Camera''' folder, and then syncthing for everything else. In my opinion, it doesn’t make sense to use Immich by itself to do this; Immich is for photos &amp; videos, it is not for Music, Documents, &amp; all the other folders on our phone. If we used syncthing for those files &amp; folders, and used Immich for photos/videos, that means we have two applications running at the same time, that do the same thing. This means 2 points of failure rather than 1.

Syncthing was designed with one purpose in mind; transfer files from device to device. I would prefer to use a tool that was designed for the job.

As a result, I am going to set up the <code>~/androidbackup/DCIM</code> folder as an '''external library''' in Immich.

<span id="attaching-zfs-pool-to-immich"></span>
==== 2.3 Attaching ZFS pool to Immich ====

See Also, remember the giant ZFS pool we created? On my setup, that’s over 100 terabytes of stuff! Much of that are old images &amp; videos that are not in my phone photo backup directory. I want to see those in Immich.

We created a Samba share for our ZFS pool so we could access it from elsewhere. I am going to create a '''read only''' samba share that is mounted on <code>~/Pictures</code>, and then set this up with Immich as a '''second external library.'''

TL;DR - Immich will have access to everything stored in your ZFS pool archive as a photo library, as well as your android phone’s photos. This allows me to perform machine learning on everything; my android phone photo backups, current android phone photos, as well as all of my stuff from the past 15 years all within one piece of software.

After this is done I will be able to use the search feature and find photos I forgot about within seconds, dating back 15 years. Awesome. :)

<blockquote>'''QUESTION:''' Why do we want the zfs pool share to be read only?

The '''androidstuff''' virtual machine that houses our syncthing backup of our android phone is going to be backed up regularly to our zfs pool. We have a copy of that being backed up every week. The entire ZFS pool, for me, is over 100 terabytes - so having version controlled backups is much more difficult.

As a result, I am personally much more protective of the data on my zfs pool than I am the data on my androidphone backups.
</blockquote>
<span id="step-3-mount-a-read-only-samba-share-of-the-zfs-pool-for-immich-onto-the-androidstuff-virtual-machine"></span>
== Step 3: Mount a read only samba share of the ZFS pool for Immich onto the androidstuff virtual machine ==

We are going to do the following:

# On the '''happycloud''' host machine, create another samba share of our ZFS pool <code>/mediapool/archive</code> that is read only.
# Mount this inside the '''androidstuff''' virtual machine on <code>~/Pictures</code> which is the Pictures subdirectory of my home directory. <code>~/</code> is shorthand for your home directory; in my case, <code>~/</code> is the same as <code>/home/louis/</code>

<span id="modify-samba-configuration-on-happycloud-host-machine"></span>
==== 3.1 Modify samba configuration on happycloud host machine ====

SSH into the happycloud host machine:

<pre>ssh louis@happycloud.home.arpa</pre>
or

<pre>ssh louis@192.168.5.2</pre>
Our <code>/etc/samba/smb.conf</code> file currently looks like this:

<pre>[global]
    # Network settings
    workgroup = HOME
    realm = home.arpa
    netbios name = happycloud
    server string = ZFS Archive Server
    dns proxy = no
    
    # Security settings
    security = user
    map to guest = bad user
    server signing = auto
    client signing = auto
    
    # Logging
    log level = 1
    log file = /var/log/samba/%m.log
    max log size = 1000
    
    # Performance optimization
    socket options = TCP_NODELAY IPTOS_LOWDELAY
    read raw = yes
    write raw = yes
    use sendfile = yes
    min receivefile size = 16384
    aio read size = 16384
    aio write size = 16384
    
    # Multichannel support
    server multi channel support = yes
    
    # Disable unused services
    load printers = no
    printing = bsd
    printcap name = /dev/null
    disable spoolss = yes
    
    # Character/Unix settings
    unix charset = UTF-8
    dos charset = CP932

[archive]
    comment = ZFS Archive Share
    path = /mediapool/archive
    valid users = louis
    invalid users = root
    browseable = yes
    read only = no
    writable = yes
    create mask = 0644
    force create mode = 0644
    directory mask = 0755
    force directory mode = 0755
    force user = louis
    force group = louis
    veto files = /._*/.DS_Store/.Thumbs.db/.Trashes/
    delete veto files = yes
    follow symlinks = yes
    wide links = no
    ea support = yes
    inherit acls = yes
    hide unreadable = yes
    guest ok = no</pre>
We are going to add something like this to the bottom of the <code>/etc/samba/smb.conf</code> file. Obviously '''feel free to set the <code>path</code> folder to what YOU want Immich to see'''. This will be read-only, so if something happens on your host, you won’t lost everything.

Use nano to edit the file:

<pre>sudo nano -w /etc/samba/smb.conf</pre>
Enter the following at the end. Hit enter so there’s a pretty little space before the new section. :)

<pre>[immich]
    comment = ZFS Archive Share (Read-Only)
    path = /mediapool/archive
    valid users = louis
    browseable = yes
    read only = yes
    guest ok = no
    create mask = 0644
    directory mask = 0755
    veto files = /._*/.DS_Store/.Thumbs.db/.Trashes/
    delete veto files = yes
    follow symlinks = yes
    wide links = no
    ea support = yes
    inherit acls = yes
    hide unreadable = yes</pre>
<span id="configure-the-samba-share-on-the-androidstuff-virtual-machine"></span>
==== 3.2 Configure the samba share on the androidstuff virtual machine ====

We want this to mount each time the '''androidstuff''' virtual machine that will run Immich boots. To do this, we will edit <code>/etc/fstab</code> - the file that defines where hard drives, partitions, network shares, are mounted on the filesystem.

We have to install the packages that allow us to mount samba shares:

<pre>sudo apt install cifs-utils -y</pre>
Edit the file:

<pre>sudo nano -w /etc/fstab</pre>
Add the following line:

<pre>//192.168.5.2/immich /home/louis/Pictures cifs ro,credentials=/etc/samba_credentials,iocharset=utf8,vers=3.0 0 0</pre>
Make sure that the IP address matches the IP address of the machine that you have your ZFS pool on.

* <code>//192.168.5.2</code> is the address of the computer that is running samba server for our samba share.
** <code>immich</code> is the name of the samba share.
** In '''happycloud'''’s’ <code>/etc/samba/smb.conf</code> configuration file, the line<code>path = /mediapool/archive</code> is present under the <code>[immich]</code> share settings.
** Therefore, `<code>//192.168.5.2/immich</code> will show us <code>/mediapool/archive</code> on the machine located at <code>192.168.5.2</code>
* <code>cifs</code> is the filesystem type. CIFS stands for '''Common Internet File System'''.
* <code>ro</code> means readonly.
* <code>/etc/samba_credentials</code> is the file that will house the username &amp; password to access this share.
* For the love of god, do not forget to set the proper permissions on the<code>/etc/samba_credentials</code> file when I tell you to.

Once you’re done adding that line to the file, we need to provide it a username/password so it can log into the password protected share.

<pre># Create the credentials file that will house the username &amp; password:
sudo nano -w /etc/samba_credentials</pre>
Add your username and password you set when you set the password for your samba user to the file in the following format:

<pre>username=louis
password=passwordman</pre>
If you forgot what the samba password is for your user, refer to '''step 6.5''' in the '''Setting up ZFS for data storage''' portion of the guide.

'''Make sure that this file is not accessible by anyone besides root!'''

<pre>sudo chown root /etc/samba_credentials
sudo chmod 600 /etc/samba_credentials</pre>
<span id="set-the-permissions-for-samba-credentials-file"></span>
==== 3.3 Set the permissions for samba credentials file ====

Important enough to be worth stating again:

<pre>sudo chown root /etc/samba_credentials
sudo chmod 600 /etc/samba_credentials</pre>

-----

<span id="mount-the-samba-share-on-the-androidstuff-virtual-machine"></span>
==== 3.4 Mount the samba share on the androidstuff virtual machine ====

Run the following to mount everything in the <code>/etc/fstab</code> file, including your samba share.

<pre>sudo mount -a
sudo systemctl daemon-reload</pre>
<span id="make-sure-it-worked."></span>
==== 3.5 Make sure it worked. ====

In <code>/home/louis/Pictures</code>on the '''androidstuff''' virtual machine you should see everything that is on <code>/mediapool/archive</code> on the '''happycloud''' host server. Try making a file and saving it there. It shouldn’t work.

Create a file on happycloud. Go to the terminal window for happycloud, or just ssh in if you don’t have one open.

<pre>ssh louis@happycloud.home.arpa

# Put a file called hello_world.log that says &quot;hi&quot; inside of it into the /mediapool/archive directory

echo &quot;hi&quot; &gt; /mediapool/archive/helloworld.log</pre>
Then, on the '''androidstuff''' virtual machine, try to view it. We mounted this samba share on <code>/home/louis/Pictures</code> so <code>hello_world.log</code> should show up on <code>/home/louis/Pictures/hello_world.log</code>

<pre>louis@androidstuff:~$ cat helloworld.log
louis@androidstuff:~$ cat ~/Pictures/helloworld.log 
hi
louis@androidstuff:~$ rm ~/Pictures/helloworld.log 
rm: remove write-protected regular file '/home/louis/Pictures/helloworld.log'? y
rm: cannot remove '/home/louis/Pictures/helloworld.log': Read-only file system
louis@androidstuff:~$ sudo rm ~/Pictures/helloworld.log 
[sudo] password for louis: 
rm: cannot remove '/home/louis/Pictures/helloworld.log': Read-only file system
</pre>
As you can see, I can see the file, I can read the file, but I can’t delete the file. Perfect.

<span id="step-4-make-your-directories"></span>
== Step 4: Make your directories ==

<span id="create-directory-structure"></span>
==== 4.1 Create Directory Structure ====

I like to put the programs I am downloading/working on in <code>/home/louis/Downloads/programs</code> The <code>~/</code> means your home directory: so if your username is chris, <code>~/Downloads/programs</code> means <code>/home/chris/Downloads/programs</code>

<pre># Create and enter directory
mkdir -p ~/Downloads/programs/immich-app
cd ~/Downloads/programs/immich-app</pre>
<span id="download-program"></span>
==== 4.2 Download Program ====

This is installed via docker and the installation files/instructions from Immich themselves are completely plug &amp; play. a

<pre># Get docker-compose.yml
wget -O docker-compose.yml https://github.com/immich-app/immich/releases/latest/download/docker-compose.yml

# Get environment file
wget -O .env https://github.com/immich-app/immich/releases/latest/download/example.env</pre>
<span id="optional-hardware-acceleration-files"></span>
==== 4.3 Optional Hardware Acceleration Files ====

I don’t use hardware acceleration since my machine does not have a GPU, or any sort of coral device. This is experimental as well so it may give you issues. However, if you plan to use hardware acceleration, grab these to set them up &amp; [https://immich.app/docs/features/ml-hardware-acceleration/ follow the instructions from Immich documentation].

<pre># For transcoding acceleration
wget -O hwaccel.transcoding.yml https://github.com/immich-app/immich/releases/latest/download/hwaccel.transcoding.yml

# For machine learning acceleration
wget -O hwaccel.ml.yml https://github.com/immich-app/immich/releases/latest/download/hwaccel.ml.yml</pre>
<span id="step-5-edit-docker-compose.yml-environment-file"></span>

== Step 5: Edit docker-compose.yml &amp; Environment File ==

<span id="edit-the-.env-file"></span>
==== 5.1 Edit the <code>.env</code> file ====

<pre>nano -w .env</pre>
<ol style="list-style-type: decimal;">
<li><p>'''Database Setting'''</p>
<ul>
<li>Change <code>DB_PASSWORD</code> . You should use characters from A to Z, a to z, and 0 to 9 - don’t use anything funky. I recommend the [https://bitwarden.com/password-generator/#password-generator Bitwarden password generator].
<ul>
<li>You can use bitwarden password generator on their website without installing their program, but I suggest installing their program at some point.</li></ul>
</li></ul>
</li>
<li><p>'''Upload Location'''</p>
<ul>
<li><p>Set <code>UPLOAD_LOCATION</code> to where you want items you upload to immich to go.</p>
<p>-I don’t use this because I use syncthing to upload things to <code>/home/louis/androidbackup</code>** rather than uploading straight to immich.</p></li>
<li><p>For the purposes of how I use Immich &amp; this guide, I will not be changing this.</p></li></ul>
</li>
<li><p>'''Timezone'''</p>
<ul>
<li>Uncomment and set the <code>TZ=</code> line to your timezone.</li>
<li>Find timezone codes [https://en.wikipedia.org/wiki/List_of_tz_database_time_zones here]</li>
<li>For example, mine would be <code>America/Chicago</code></li></ul>
</li></ol>

<span id="edit-the-docker-compose.yml-file"></span>
==== 5.2 Edit the <code>docker-compose.yml</code> file ====

Open the file for editing:

<pre>nano -w docker-compose.yml</pre>
This file in its entirety is fine as is. Nothing has to be changed. The two lines I add are to allow immich access to the <code>~/Pictures</code> directory where my ZFS pool’s files are located, and the <code>~/androidbackup/DCIM</code> directory where the photos &amp; videos I took using the camera app on my android phone are stored.

The two lines I added to the file below are:

<pre>      - /home/louis/androidbackup/DCIM:/files/phonepics:rw
      - /home/louis/Pictures:/files/zfspics:ro</pre>
These lines do the following:

* Makes <code>/home/louis/androidbackup/DCIM</code> on the host computer Immich is running on show up as <code>/files/phonepics</code> inside the docker container for Immich, with read write permissions.
* Makes <code>/home/louis/Pictures</code> on the host computer Immich is running on show up as <code>/files/zfspics</code> inside the docker container for Immich, with read only permissions.

To see where I put these in the context of the full file, look below:

<pre>#
# WARNING: Make sure to use the docker-compose.yml of the current release:
#
# https://github.com/immich-app/immich/releases/latest/download/docker-compose.yml
#
# The compose file on main may not be compatible with the latest release.
#

name: immich

services:
  immich-server:
    container_name: immich_server
    image: ghcr.io/immich-app/immich-server:${IMMICH_VERSION:-release}
    # extends:
    #   file: hwaccel.transcoding.yml
    #   service: cpu # set to one of [nvenc, quicksync, rkmpp, vaapi, vaapi-wsl] for accelerated transcoding
    volumes:
      # Do not edit the next line. If you want to change the media storage location on your system, edit the value of UPLOAD_LOCATION in the .env file
      - ${UPLOAD_LOCATION}:/usr/src/app/upload
      - /etc/localtime:/etc/localtime:ro
      - /home/louis/androidbackup/DCIM:/files/phonepics:rw
      - /home/louis/Pictures:/files/zfspics:ro
    env_file:
      - .env
    ports:
      - '2283:2283'
    depends_on:
      - redis
      - database
    restart: always
    healthcheck:
      disable: false

  immich-machine-learning:
    container_name: immich_machine_learning
    # For hardware acceleration, add one of -[armnn, cuda, openvino] to the image tag.
    # Example tag: ${IMMICH_VERSION:-release}-cuda
    image: ghcr.io/immich-app/immich-machine-learning:${IMMICH_VERSION:-release}
    # extends: # uncomment this section for hardware acceleration - see https://immich.app/docs/features/ml-hardware-acceleration
    #   file: hwaccel.ml.yml
    #   service: cpu # set to one of [armnn, cuda, openvino, openvino-wsl] for accelerated inference - use the `-wsl` version for WSL2 where applicable
    volumes:
      - model-cache:/cache
    env_file: 
      - .env
    restart: always
    healthcheck:
      disable: false

  redis:
    container_name: immich_redis
    image: docker.io/redis:6.2-alpine@sha256:2ba50e1ac3a0ea17b736ce9db2b0a9f6f8b85d4c27d5f5accc6a416d8f42c6d5
    healthcheck:
      test: redis-cli ping || exit 1
    restart: always

  database:
    container_name: immich_postgres
    image: docker.io/tensorchord/pgvecto-rs:pg14-v0.2.0@sha256:90724186f0a3517cf6914295b5ab410db9ce23190a2d9d0b9dd6463e3fa298f0
    environment:
      POSTGRES_PASSWORD: ${DB_PASSWORD}
      POSTGRES_USER: ${DB_USERNAME}
      POSTGRES_DB: ${DB_DATABASE_NAME}
      POSTGRES_INITDB_ARGS: '--data-checksums'
    volumes:
      # Do not edit the next line. If you want to change the database storage location on your system, edit the value of DB_DATA_LOCATION in the .env file
      - ${DB_DATA_LOCATION}:/var/lib/postgresql/data
    healthcheck:
      test: pg_isready --dbname='${DB_DATABASE_NAME}' --username='${DB_USERNAME}' || exit 1; Chksum=&quot;$$(psql --dbname='${DB_DATABASE_NAME}' --username='${DB_USERNAME}' --tuples-only --no-align --command='SELECT COALESCE(SUM(checksum_failures), 0) FROM pg_stat_database')&quot;; echo &quot;checksum failure count is $$Chksum&quot;; [ &quot;$$Chksum&quot; = '0' ] || exit 1
      interval: 5m
      start_interval: 30s
      start_period: 5m
    command:
      [
        'postgres',
        '-c',
        'shared_preload_libraries=vectors.so',
        '-c',
        'search_path=&quot;$$user&quot;, public, vectors',
        '-c',
        'logging_collector=on',
        '-c',
        'max_wal_size=2GB',
        '-c',
        'shared_buffers=512MB',
        '-c',
        'wal_compression=on',
      ]
    restart: always

volumes:
  model-cache:</pre>
<blockquote>'''DOCKER CHEAT SHEET: going through <code>docker-compose.yml</code> file for Immich'''
This file sets up a bunch of containers''(virtualized, minimalistic computers that run inside your computer)'' for the Immich photo gallery/library/machine learning &amp; management system.

'''1. <code>name: immich</code>'''
This is the name of the overall Docker Compose project.

'''2. <code>services:</code>'''
This section lists all the containers (services) that make up the Immich application. Each service is a part of the overall program.

'''immich-server''' '''3. <code>immich-server:</code>'''
This is the primary backend service of Immich. It handles the main functions of the program like uploading, managing, &amp; displaying photos.

'''4. <code>container_name: immich_server</code>'''
This is the name of the container so when you run <code>docker ps -a</code> to see what containers are running you can see this one and know what it is for immediately. Custom name for the main immich container so it is easy to find when you type <code>docker ps -a</code> . Sometimes while debugging things that are not working you may want to enter the environment of the virtual container''(this is like sshing into your server, but into the virtual server that runs immich)'', which you can do by typing <code>docker exec -it immich_server bash</code> - but to do that you need to know which container is which! This is where using sensible names comes into play.

'''5. <code>image: ghcr.io/immich-app/immich-server:${IMMICH_VERSION:-release}</code>'''
This tells it what Docker image to use for the backend. It pulls the latest stable version unless you’ve overridden <code>IMMICH_VERSION</code> in your <code>.env</code> file. Since Immich does not destroy their software with new releases, I am setting it to grab the latest version.

'''6. <code>volumes:</code>'''

* <code>${UPLOAD_LOCATION}:/usr/src/app/upload</code>: Links the photo upload storage location from your system to the container. The path <code>${UPLOAD_LOCATION}</code> is defined in the <code>.env</code> file. Whatever this is will show up inside the container at <code>/usr/src/app/upload</code>
* <code>/etc/localtime:/etc/localtime:ro</code>: This makes the container use the same time as your computer’s time. The <code>:ro</code> makes it read-only so your computer can’t do what the characters in predestination did. The only thing worse than using google photos is '''''SPOILER ALERT''''' having your machine send you back in time so you are an orphan who was its own mother like in '''Predestination'''. Still a decent time travel movie but it has nothing on '''Primer'''.
* <code>/home/louis/androidbackup/DCIM:/files/phonepics:rw</code>: Maps a directory with phone pictures to <code>/files/phonepics</code> in the container. This is read-write (<code>rw</code>). SO whatever is inside my <code>/home/louis/androidbackup/DCIM</code> directory on the <code>androidstuff</code> virtual machine running at <code>192.168.5.5</code> that we set up will show up inside the <code>immich-server</code> docker container under the directory <code>/files/phonepics</code>.
* <code>/home/louis/Pictures:/files/zfspics:ro</code>: Maps a directory with other pictures to <code>/files/zfspics</code> in the container. This one is read-only (<code>ro</code>).

'''7. <code>env_file:</code>'''
Loads environment variables from the <code>.env</code> file, which centralizes configuration settings.

'''8. <code>ports:</code>'''

* <code>'2283:2283'</code>: Maps port <code>2283</code> on your host system to port <code>2283</code> in the container. This allows you to access Immich’s server on your browser at <code>http://192.168.5.5:2283</code> since we are installing this dockerized deployment of Immich to the <code>androidstuff</code> virtual machine located at <code>192.168.5.5</code>

'''9. <code>depends_on:</code>'''
This lists the services this container depends on. <code>redis</code> and <code>database</code> must be running before the server starts. Don’t be scared by the word depends. It is included in ''“dependency”'', but you’re using a docker image deployed by good developers; dependencies are no longer something to be afraid of :) I promise :)

'''10. <code>restart: always</code>'''
Automatically restarts the container if it crashes or if the system reboots. When you turn the system on immich will be on without having to go to its directory &amp; run <code>docker compose up -d</code> each time the computer starts.

'''11. <code>healthcheck:</code>'''
Monitors the container’s health. The <code>disable: false</code> line means health checks are enabled.

'''immich-machine-learning''' '''12. <code>immich-machine-learning:</code>'''
This container handles machine learning tasks, like face or object recognition''(searching for “cat on chair”)'' in your photos.

'''13. <code>container_name: immich_machine_learning</code>'''
Custom name for the machine learning container so it is easy to find when you type <code>docker ps -a</code> . Sometimes while debugging things that are not working you may want to enter the environment of the virtual container''(this is like sshing into your server, but into the virtual server that runs immich)'', which you can do by typing <code>docker exec -it immich_machine_learning bash</code> - but to do that you need to know which container is which! This is where using sensible names comes into play.

'''14. <code>image:</code>'''
Pulls the machine learning image from GitHub. You can enable hardware acceleration by adding a specific tag (e.g., <code>-cuda</code>) if supported by your system.

'''15. <code>volumes:</code>'''

* <code>model-cache:/cache</code>: Links a Docker-managed volume to the container’s <code>/cache</code> directory for storing machine learning model data.

'''16. <code>env_file:</code>'''
Loads environment variables from <code>.env</code> for consistent configuration. For instance, instead of editing certain configuration files after or while setting up/compiling the program, you put them in the environment file and when the docker container starts, it uses what is in the environment file.

'''17. <code>restart: always</code>'''
The container restarts if it crashes &amp; will start up with the computer.

'''18. <code>healthcheck:</code>'''
Keeps the container healthy and ensures it’s running properly.

'''redis''' '''19. <code>redis:</code>'''
Redis is a high-speed database used for caching data and managing background tasks.

'''20. <code>container_name: immich_redis</code>'''
Custom name for the Redis container so it is easy to find when you type <code>docker ps -a</code> . Sometimes while debugging things that are not working you may want to enter the environment of the virtual container''(this is like sshing into your server, but into the virtual server that runs immich)'', which you can do by typing <code>docker exec -it immich_redis bash</code> - but to do that you need to know which container is which! This is where using sensible names comes into play.

'''21. <code>image:</code>'''
Specifies the exact Redis image to use, including a SHA256 checksum for security.

'''22. <code>healthcheck:</code>'''
Runs a simple test (<code>redis-cli ping</code>) to confirm the Redis service is working.

'''23. <code>restart: always</code>'''
Automatically restarts Redis if it fails/it starts with the computer.

'''database''' '''24. <code>database:</code>'''
This is the PostgreSQL database, which stores metadata and application data for Immich.

'''25. <code>container_name: immich_postgres</code>'''
Custom name for the database container so it is easy to find when you type <code>docker ps -a</code> . Sometimes while debugging things that are not working you may want to enter the environment of the virtual container''(this is like sshing into your server, but into the virtual server that runs immich)'', which you can do by typing <code>docker exec -it immich_postgres bash</code> - but to do that you need to know which container is which! This is where using sensible names comes into play.

'''26. <code>image:</code>'''
Specifies a custom PostgreSQL image with vector support, used by Immich for advanced search features.

'''27. <code>environment:</code>'''
- <code>POSTGRES_PASSWORD</code>: Password for the database.
- <code>POSTGRES_USER</code>: Username for the database.
- <code>POSTGRES_DB</code>: Name of the database.
- <code>POSTGRES_INITDB_ARGS</code>: Additional arguments for database

'''28. <code>volumes:</code>'''

* <code>${DB_DATA_LOCATION}:/var/lib/postgresql/data</code>: Maps the database storage location from your system to the container. Edit <code>${DB_DATA_LOCATION}</code> in the <code>.env</code> file to change where your database files are stored.

'''29. <code>healthcheck:</code>'''
Runs periodic checks to ensure the database is healthy. It verifies that the database is running, accessible, and free of checksum errors.

'''30. <code>command:</code>'''
Customizes PostgreSQL’s behavior with specific options, like enabling vector indexing (<code>shared_preload_libraries=vectors.so</code>) &amp; improving performance with optimized settings like <code>max_wal_size=2GB</code>.

'''31. <code>restart: always</code>'''
Makes database container restart if something goes wrong/it starts with the computer.

'''volumes''' '''32. <code>volumes:</code>'''
- <code>model-cache</code>: A named volume for storing machine learning models. This ensures that cached data persists across container restarts or recreations.
</blockquote>
<span id="step-5-start-the-system"></span>
== Step 5: Start the System ==

While in the directory you downloaded the <code>docker-compose.yml</code> and <code>.env</code> file to, run the following:

<pre>docker compose up</pre>
I like to type <code>docker compose up</code> at first without the <code>-d</code> because I can see what is happening without having to use tail on a logfile somewhere. If you don’t care to do that, you can start it up like this with the <code>-d</code> which allows the program to start without it stopping when you close the terminal window you ran the command in.

<pre>docker compose up -d</pre>
'''Visiting Immich web interface''': at this point you should be able to visit <code>http://192.168.5.5:2283</code> or <code>http://androidstuff.home.arpa:2283</code> and see Immich, in all its glory :)

<span id="if-it-doesnt-work"></span>
== If it doesn’t work: ==

# '''Wrong Docker Version''' If you get <code>unknown shorthand flag: 'd' in -d</code>, you’re likely using the wrong Docker version. Fix by:
#* Remove the distribution’s docker.io package. If you used snap, I will hurt you.
#* Install Docker from the official repository
#* If you used ubuntu version of docker installed via snap upon installation of ubuntu server after all the times I told you not to in the past 1000 pages of this guide…..
#* You asked for this.
# '''Docker Compose Command'''
#* Use <code>docker compose</code> (not <code>docker-compose</code>)
#* Installing from Docker official repository is required here. You saw how to do this in the onlyoffice setup section on this virtual machine.

<div class="figure">

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:image-20241114081931118.png
</gallery>

</div>
<div class="figure">

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:image-20241114081943465.png
</gallery>

</div>
<div class="figure">

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:image-20241114081947837.png
</gallery>

</div>
<div class="figure">

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:image-20241114081952635.png
</gallery>

</div>
<div class="figure">

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:image-20241114082104266.png
</gallery>

</div>
<div class="figure">

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:image-20241114082148529.png
</gallery>

</div>
<div class="figure">

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:image-20241114082040918.png
</gallery>

</div>
<div class="figure">

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:image-20241114082203688.png
</gallery>

</div>
<div class="figure">

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:image-20241114082215711.png
</gallery>

</div>
<div class="figure">

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:image-20241114082308766.png
</gallery>

</div>
<div class="figure">

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:image-20241114082636100.png
</gallery>

</div>
<div class="figure">

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:image-20241114082702362.png
</gallery>

</div>
<div class="figure">

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:image-20241114082732250.png
</gallery>

</div>
<div class="figure">

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:image-20241114083349204.png
</gallery>

</div>
<div class="figure">

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:image-20241114102736005.png
</gallery>

</div>
<div class="figure">

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:image-20241114085055146.png
</gallery>

</div>
<div class="figure">

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:image-20241114085114634.png
</gallery>

</div>
<div class="figure">

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:image-20241114085134227.png
</gallery>

</div>
<div class="figure">

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:image-20241114085158141.png
</gallery>

</div>
<div class="figure">

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:image-20241114085215897.png
</gallery>

</div>
<div class="figure">

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:image-20241114085323218.png
</gallery>

</div>
<div class="figure">

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:image-20241114085419575.png
</gallery>

</div>
<div class="figure">

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:image-20241114102843677.png
</gallery>

</div>
<div class="figure">

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:image-20241114085501334.png
</gallery>

</div>
<div class="figure">

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:image-20241114103219273.png
</gallery>

</div>
<div class="figure">

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:image-20241114085653523.png
</gallery>

</div>
<div class="figure">

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:image-20241114085749209.png
</gallery>

</div>
<div class="figure">

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:image-20241114085900361.png
</gallery>

</div>
<div class="figure">

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:image-20241114085939405.png
</gallery>

</div>
<div class="figure">

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:image-20241114085956763.png
</gallery>

</div>
<div class="figure">

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:image-20241114090009152.png
</gallery>

</div>
<div class="figure">

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:image-20241114090053347.png
</gallery>

</div>
<div class="figure">

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:image-20241114103825099.png
</gallery>

</div>
<div class="figure">

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:image-20241114104024871.png
</gallery>

</div>
<span id="step-6-configure-immich"></span>
== Step 6: Configure Immich ==

Once it’s started it’ll ask you to set up a username and a password. Once that’s done, we have a few tasks to complete.

<span id="set-up-your-android-backup-zfs-pool-as-libraries-in-immich"></span>
==== 6.1 Set up your android backup &amp; zfs pool as libraries in Immich ====

This is necessary so you can see your files.

<ol style="list-style-type: decimal;">
<li><p>Click the circle in the upper right corner that has the first letter of your username.</p></li>
<li><p>Click '''Administration'''</p></li>
<li><p>Click '''External Libraries''' on the left menu</p></li>
<li><p>Click on the plus or on the '''Create Library''' button in the upper right to create a library.</p></li>
<li><p>Create two libraries: and set yourself as the owner of each.</p></li>
<li><p>Click on the three dots next to the library.</p></li>
<li><p>Click '''Rename'''</p></li>
<li><p>Name each library - (e.g. '''zfs pool''' and '''android phone'''.)</p></li>
<li><p>Click the three dots again and click '''Edit Import Paths'''</p></li>
<li><p>Set each external library to have the path we chose above for our zfs pool and our android phone backup. <code>- /home/louis/androidbackup/DCIM:/files/phonepics:rw       - /home/louis/Pictures:/files/zfspics:ro</code></p></li>
<li><p>Once done with this, go back to '''Settings''' in the left hand menu</p></li>
<li><p>Go to '''Video Transcoding Settings'''</p></li>
<li><p>If you want video proxies created so you are watching lower bitrate files when you load immich(useful if you use this on a phone with bad internet speeds), change '''Transcode policy''' to '''All videos'''</p>
<blockquote><p>'''NOTE:''' Transcoding videos doesn’t delete the original. It creates new videos in a subfolder of the <code>immich-app</code> directory. The original video file is preserved in full quality in its original location.</p></blockquote></li>
<li><p>If you have a fast computer, or lots of patience, set '''Preset''' to '''fast''' - this will make video files that are smaller for the same quality as '''ultrafast'''. For '''Constant Rate Factor''', higher is smaller file/worse quality, lower number is larger file/better quality. If you are making video proxies because your internet service sucks I’d set this to 28.</p></li>
<li><p>In '''Settings''', go over to '''External Library'''</p></li>
<li><p>Under '''Library Watching''' enable '''Watch external libraries for file changes'''</p></li>
<li><p>Under '''Periodic Scanning''' Make sure this is turned on. I would make this something daring; perhaps once an hour. Remember, since we are not using the Immich app to upload the photos to Immich, Immich is not aware without scanning manually if we have added files or not.</p></li>
<li><p>On the left hand menu, go over to '''Jobs'''.</p></li>
<li><p>Next to '''LIBRARY''', click the ALL button.</p></li>
<li><p>Wait patiently.</p></li>
<li><p>You’re done. :)</p></li></ol>

<span id="step-7-enjoy-immich"></span>

== Step 7: Enjoy Immich ==

Once the '''Jobs''' tab shows that Immich is done processing everything, head over to the homepage, and try the search box. It’s awesome.

<div class="figure">

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:image-20241114111713397.png
</gallery>

</div>
<div class="figure">

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:image-20241114111737760.png
</gallery>

</div>
<div class="figure">

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:image-20241114111751348.png
</gallery>

</div>
<div class="figure">

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:image-20241114112125953.png
</gallery>

</div>
<div class="figure">

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:image-20241114112154974.png
</gallery>

</div>
<div class="figure">

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:image-20241114114756999.png
</gallery>

</div>
<div class="figure">

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:image-20241114115240120.png
</gallery>

</div>
<div class="figure">

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:image-20241114115256590.png
</gallery>

</div>
<span id="step-8-install-android-app"></span>
== Step 8: Install Android App ==

<span id="install-the-f-droid-store-app"></span>
==== 8.1 Install the F-Droid store app ====

Download F-Droid from the [https://f-droid.org/docs/Get_F-Droid/ F-Droid website] and then open the apk to install it. F-Droid allows you to downlod all sorts of interesting open source apps.

<span id="install-immich"></span>
==== 8.2 Install Immich ====

Find &amp; install Immich

<span id="start-immich"></span>
==== 8.3 Start Immich ====

When you start Immich, in the '''Server Endpoint URL''' field, but the same thing you put in your web browser to connect; <code>http://192.168.5.5:2283</code> or <code>http://androidstuff.home.arpa:2283</code>

Don’t forget to put the port. Also, this will only work on local wifi or with your VPN on from your smartphone. '''Make sure you are connected to wifi or are connected to the VPN!'''

<span id="notes-on-upgradesupdates"></span>
=== Notes on upgrades/updates: ===

* ''“Breaking changes”'' are when an old version of Immich will not work properly when updating to a new version of immich.
* Review release notes to see if this is the case with your version. This is something that is being worked on so it won’t happen in the future. Alex is great with informing users on these changes.

<span id="update-process"></span>
=== Update Process ===

To upgrade to a new version, go to the directory with Immich, in our case, <code>~/Downloads/programs/immich-app</code>. Turn Immich off, pull the new version, and then turn it on again. '''I suggest having a backup of everything before doing this.''' Doing perfect VM backups will be in the next section.

<pre>cd ~/Downloads/programs/immich-app
docker compose down
docker compose pull
docker compose up -d</pre>
<span id="nextcloud-notes-to-replace-google-keep"></span>
= Nextcloud Notes to replace Google Keep =

For most intents &amp; purposes, nextcloud is horrible. It does one thing right for me; notes. plaintext, or markdown notes.

I live my life on a schedule where my day is mapped out in 5 to 15 minute increments, that is constantly changing. I [https://www.youtube.com/watch?v=JKCSLur0VYw discussed this in a video 11 years ago]. Throughout the day I am constantly opening my notes application &amp; hitting the voice-to-text button so I can talk into my phone before I forget what I wanted to type or do. Sometimes, in the middle of the note I forget what I wanted to jot down and will speak out something that resembles the idea I hope I remember later.

I need my notes. I need them to be easily accessible, available as either lists or as post-it-notes in the style of google keep. I need the notes application and the web interface to be easily accessible without having to install extra stuff on my computer if I don’t want to. I need the interface to be as simplistic &amp; uncluttered as possible. More options = more chances for confusion for someone who ''needs'' a notes application(or physical notepad) to not forget what I am doing constantly.

Nextcloud’s interface does that for me. It mimics google keep’s functionality and is the closest spot on thing I’ve found to it.

'''If you want something that is well programmed, forget about this. Go install Joplin. I use nextcloud notes because the interface &amp; ease of use/deployment is worth it for me.''' I have played around with joplin. It’s obviously better coded software; but the phone application interface isn’t it for me, and I don’t want to go hunting for a client that will at best provide me the same experience I already have.

I am a single user loading plain text files. As bad as nextcloud is, it can’t mess that up. Well, maybe it can - but it hasn’t for me yet.

Follow these steps to deploy Nextcloud on your server (IP: <code>192.168.5.5</code>) with Docker Compose. This setup is restricted to clients within the <code>192.168.5.0/24</code> and <code>192.168.6.0/24</code> subnets.

<span id="installing-nextcloud-for-notes"></span>
== Installing Nextcloud for notes ==

Nextcloud notes we install via docker. We will install ONLY the notes component when we enter the web interface so the least amount of nextcloud is on our system as is necessary.

<span id="step-1-ssh-into-the-androidstuff-virtual-machine-computer"></span>
=== Step 1: SSH into the androidstuff virtual machine computer ===

<pre>ssh louis@192.168.5.5</pre>
OR

<pre>ssh louis@androidstuff.home.arpa</pre>
<span id="step-2-install-docker"></span>
=== Step 2: Install docker ===

<span id="verify-docker-installation-1"></span>
==== 2.1 Verify Docker installation: ====

<span id="if-you-elected-to-install-immich-or-onlyoffice-on-this-virtual-machine-this-part-is-already-done-you-can-skip-to-step-3"></span>
===== IF YOU ELECTED TO INSTALL IMMICH OR ONLYOFFICE ON THIS VIRTUAL MACHINE, THIS PART IS ALREADY DONE &amp; YOU CAN SKIP TO STEP 3! =====

**If you installed onlyoffice or immich on the androidstuff virtual machine, &amp; followed the instructions for it, you already installed docker properly on this virtual machine, and have no need to do this again. Skip to step 3 if that is the case.

Run <code>docker --version</code> and make sure the version is 24.0.0 or later. If not, remove the old version:

<pre>sudo apt remove docker docker-engine docker.io containerd runc</pre>
<span id="install-docker-using-official-docker-script-4"></span>
==== 2.3 Install Docker using official Docker script: ====

<pre>curl -fsSL https://get.docker.com -o get-docker.sh
sudo sh get-docker.sh</pre>
<blockquote>'''Note:''' It’s very important to use the official Docker installation and not the Snap version. The Snap version can cause issues due to its sandboxed nature, making it a mess for <code>mailcow</code>’s requirements. It is bad for our purposes, don’t use it.
</blockquote>
<span id="install-docker-compose-prerequisites-1"></span>
==== 2.4 Install Docker Compose &amp; prerequisites: ====

<pre>sudo apt install docker-compose-plugin -y
sudo systemctl enable --now docker</pre>
<span id="make-sure-it-worked-1"></span>
==== 2.5 Make sure it worked ====

Run <code>docker compose version</code> and make sure the version is 2.0 or higher.

<span id="step-3-install-nextcloud-using-docker"></span>
=== Step 3: Install nextcloud using docker ===

<span id="create-directory-to-store-docker-compose-file-volumes"></span>
==== 3.1 Create directory to store Docker Compose file &amp; volumes: ====

<pre>mkdir -p ~/nextcloud &amp;&amp; cd ~/nextcloud</pre>
<span id="copy-your-docker-compose.yml-file-into-this-directory-or-create-it"></span>
==== 3.2 Copy your <code>docker-compose.yml</code> file into this directory or create it: ====

<pre>nano docker-compose.yml</pre>
'''Paste the content below:'''

<pre>services:
  db:
    image: mariadb:10.11
    restart: always
    command: --transaction-isolation=READ-COMMITTED --log-bin=binlog --binlog-format=ROW
    volumes:
      - db:/var/lib/mysql
    environment:
      - MYSQL_ROOT_PASSWORD=rootpasswd
      - MYSQL_PASSWORD=dbpasswd
      - MYSQL_DATABASE=nextcloud
      - MYSQL_USER=nextcloud

  redis:
    image: redis:alpine
    restart: always

  app:
    image: nextcloud
    restart: always
    ports:
      - 8089:80
    depends_on:
      - redis
      - db
    volumes:
      - nextcloud:/var/www/html
    environment:
      - MYSQL_PASSWORD=dbpasswd
      - MYSQL_DATABASE=nextcloud
      - MYSQL_USER=nextcloud
      - MYSQL_HOST=db
      - NEXTCLOUD_TRUSTED_DOMAINS=192.168.5.5 192.168.5.0/24 192.168.6.0/24

volumes:
  nextcloud:
  db:</pre>
'''Save and exit the file.'''

I’ll help reformat this markdown to ensure each line starts with “&gt;” and remove the horizontal rules (“—”) while preserving all the original text exactly. Here’s the reformatted version:

<blockquote>'''DOCKER CHEAT SHEET: going over the <code>docker-compose.yml</code> for nextcloud'''
This file sets up three services (containers): one for the Nextcloud app, one for the database (MariaDB), &amp; one for caching (Redis). Let’s go through it line by line so you understand what’s going on.

'''1. <code>services:</code>'''
This section lists the containers (services) that make up the Nextcloud deployment. Each container plays a specific role in the overall application.

'''Database (<code>db</code>)'''

'''2. <code>db:</code>'''
This is the MariaDB database container. MariaDB is a database similar to mysql database. It’s where nextcloud stores info on users, settings, files, etc.

'''3. <code>image: mariadb:10.11</code>'''
This tells Docker to use the MariaDB 10.11 image. It’s a specific version of MariaDB that ensures compatibility with the version of Nextcloud you’re running. This is why docker is awesome; this just pulls the right version of the right program. You don’t have to worry about this. The maintainers of the software provide template <code>docker-compose.yml</code> files that rarely need more than minimal adjustment to work for your needs. No dependency rabbit hole to hell.

'''4. <code>restart: always</code>'''
Makes the database container restart automatically if it crashes or when the system reboots, and has it start up when you turn on the virtual machine(or computer, if you are installing directly onto the host machine)

'''5. <code>command: --transaction-isolation=READ-COMMITTED --log-bin=binlog --binlog-format=ROW</code>'''
Customizes how MariaDB runs:
- <code>--transaction-isolation=READ-COMMITTED</code>: Prevents dirty reads, ensuring reliable database transactions.
- <code>--log-bin=binlog</code>: Enables binary logging for replication (useful for backups or scaling).
- <code>--binlog-format=ROW</code>: Logs changes at the row level for better replication accuracy.

'''6. <code>volumes:</code>'''

* <code>db:/var/lib/mysql</code>: Maps the container’s <code>/var/lib/mysql</code> directory (where the database stores its files) to the <code>db</code> volume. This makes data persist even if the container is removed or restarted as it is stored to a volume(remember containers are like linux livecds, nothing is saved when you reboot them)

'''7. <code>environment:</code>'''
These environment variables configure MariaDB:
- <code>MYSQL_ROOT_PASSWORD=rootpasswd</code>: Sets the root password for MariaDB. - <code>MYSQL_PASSWORD=dbpasswd</code>: Password for the <code>nextcloud</code> user, who will access the database.
- <code>MYSQL_DATABASE=nextcloud</code>: Creates a database named <code>nextcloud</code> during container setup.
- <code>MYSQL_USER=nextcloud</code>: Creates a database user named <code>nextcloud</code>.

'''Redis (<code>redis</code>)'''

'''8. <code>redis:</code>'''
This is the Redis container which is a caching system that speeds up Nextcloud by temporarily storing frequently used data. “Speeds up” in the theoretical sense. Nothing speeds up nextcloud.

'''9. <code>image: redis:alpine</code>'''
Specifies the Redis image to use. The <code>alpine</code> tag uses a lightweight version of Redis for minimal resource usage.

'''10. <code>restart: always</code>'''
Automatically restarts the Redis container if it crashes or when the system reboots.

'''Nextcloud Application (<code>app</code>)'''

'''11. <code>app:</code>'''
This is the main container for the Nextcloud application. It provides the web interface and handles user requests.

'''12. <code>image: nextcloud</code>'''
Tells Docker to use the official Nextcloud image.

'''13. <code>restart: always</code>'''
Ensures the Nextcloud container restarts if it crashes or when the system reboots.

'''14. <code>ports:</code>'''

* <code>8089:80</code>: Maps port 80 in the container (Nextcloud’s default web server port) to port 8089 on the host. You’ll access Nextcloud in your browser at <code>http://192.168.5.5:8089</code> since this is being set up on the <code>androidstuff</code> virtual machine.

'''15. <code>depends_on:</code>'''
Ensures that <code>redis</code> and <code>db</code> containers start before the Nextcloud container. Without this, Nextcloud would crash while waiting for its database and caching system.

'''16. <code>volumes:</code>'''

* <code>nextcloud:/var/www/html</code>: Links the container’s <code>/var/www/html</code> directory (where Nextcloud’s files live) to the <code>nextcloud</code> volume. This ensures Nextcloud’s data persists even if the container is recreated.

'''17. <code>environment:</code>'''
Configures the Nextcloud container with the following environment variables:
- <code>MYSQL_PASSWORD=dbpasswd</code>: Matches the database user’s password set in the <code>db</code> service.
- <code>MYSQL_DATABASE=nextcloud</code>: Specifies the name of the database created in the <code>db</code> service.
- <code>MYSQL_USER=nextcloud</code>: Specifies the database user created in the <code>db</code> service.
- <code>MYSQL_HOST=db</code>: Tells Nextcloud where to find the database (the <code>db</code> service within this <code>docker-compose.yml</code>).
- <code>NEXTCLOUD_TRUSTED_DOMAINS=192.168.5.5 192.168.5.0/24 192.168.6.0/24</code>: Lists IP addresses or subnets that are allowed to access the Nextcloud instance. I want nextcloud to be accessible when I am on my LAN which is the same network as nextcloud, and I also want it to be accessible when I am connecting to my home server using my VPN, so I have put my LAN of <code>192.168.5.0/24</code> &amp; my VPN network of <code>192.168.6.0/24</code>

'''Volumes'''

'''18. <code>volumes:</code>'''
Defines persistent storage for Nextcloud and MariaDB:
- <code>nextcloud</code>: Stores Nextcloud’s files.
- <code>db</code>: Stores MariaDB’s database files.

'''FINAL NOTE:'''
This <code>docker-compose.yml</code> file sets up a fully functional Nextcloud deployment with three containers working together:
- '''MariaDB (db):''' Handles data storage for Nextcloud.
- '''Redis (redis):''' Speeds up Nextcloud by caching frequently used data.
- '''Nextcloud (app):''' Provides the web interface and file management.
The volumes ensure your data persists, and the environment variables make configuration easy. By using this file, you avoid dependency hell and can back up your Nextcloud setup easily by saving the volumes and <code>docker-compose.yml</code> file.
</blockquote>
<span id="deploy-the-containers"></span>
==== 3.4 Deploy the Containers ====

Run Docker Compose to start nextcloud:

<pre>docker-compose up -d</pre>
<div class="figure">

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:image-20241202040521824.png
</gallery>

</div>
<div class="figure">

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:image-20241202040603166.png
</gallery>

</div>
<div class="figure">

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:image-20241202040734008.png
</gallery>

</div>
<div class="figure">

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:image-20241202040816607.png
</gallery>

</div>
<span id="access-nextcloud-for-first-time"></span>
== Access Nextcloud for first time ==

Visit http://192.168.5.5:8089 in your web browser to complete the setup. '''Don’t enable ANY application when asked besides notes!''' Click onto the notes tab at the top to experiment with notes.

<div class="figure">

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:image-20241202041129391.png
</gallery>

</div>
<div class="figure">

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:image-20241202041140622.png
</gallery>

</div>
<div class="figure">

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:image-20241202041439357.png
</gallery>

</div>
<div class="figure">

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:image-20241202042213145.png
</gallery>

</div>
<div class="figure">

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:image-20241202041522822.png
</gallery>

</div>
<div class="figure">

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:image-20241202041531856.png
</gallery>

</div>
<div class="figure">

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:image-20241202041613763.png
</gallery>

</div>
<div class="figure">

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:image-20241202041647018.png
</gallery>

</div>
<div class="figure">

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:image-20241202041654059.png
</gallery>

</div>
<div class="figure">

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:image-20241202041700123.png
</gallery>

</div>
<span id="installing-nextcloud-android-app"></span>
== Installing Nextcloud Android App ==

I use nextcloud notes from my phone all the time. It is one of my favorite ways of getting random things I type/copy &amp; paste/dump onto my desktop into my phone &amp; vice versa. Here’s how to install the Nextcloud app on your phone and connect it to your server.


-----

<span id="step-1-install-the-nextcloud-app"></span>
=== Step 1: Install the Nextcloud App ===

# '''Open the Google Play Store''' (or F-Droid store).
# Search for '''“Nextcloud”''' and install the official app by ''“Nextcloud”''.
# Once installed, open it.
# I hope this part is self explanatory by now.


-----

<span id="step-2-add-your-server"></span>
=== Step 2: Add Your Server ===

# On the app’s welcome screen, tap '''“Log in”'''.
# Enter your server address:
<code>http://192.168.5.5:8089</code>
#* (Make sure your phone is connected to the same network as your server.
#* If not, connect to your VPN using the OpenVPN application we set up.
# Tap '''“Next”''' &amp; wait for the app to verify the server connection. It might take a while; this is nextcloud, after all.


-----

<span id="step-3-log-in"></span>
=== Step 3: Log In ===

# Enter the username &amp; password you created during the first step of accessing Nextcloud’s web interface from the web browser on your desktop earlier.
# Tap '''“Log in”'''.
# Allow the application the permission it asks for to access your nextcloud account.


-----

<span id="step-4-enable-notes-synchronization"></span>
=== Step 4: Enable Notes Synchronization ===

# Once logged in, you’ll see a list of notes.


-----

You’re done. You can write down your notes on your phone &amp; they’ll sync instantly with your server at home. You can make it look like google keep if you want. It just makes sense right out of the box with a very intuitive user interface and doesn’t try to add a bunch of stuff I don’t need/want. It works. Even though it’s nextcloud; it works. :) If the lack of https/ssl bothers you, feel free to follow the instructions from the frigate part of the guide that goes over setting up nginx as a reverse proxy so you can use ssl. If you are using onlyoffice on port 443, you’ll have to choose a different port for nextcloud, but that’s fine. You’d visit https://192.168.5.5:444 to get to nextcloud instead of https://192.168.5.5 - you’ll live!

<span id="setting-up-trusted-untrusted-wifi-with-tp-link-eap610-pfsense"></span>
= Setting Up trusted &amp; untrusted WiFi with TP-Link EAP610 &amp; pfSense =

<span id="step-1-understanding-the-problem.-why-do-this"></span>
== Step 1: Understanding the problem. Why do this? ==

Let’s say there’s a device on your network you don’t trust. You want to use it, but you don’t trust it. Exhibit A, a Chinese security camera. Hikvision makes good, cheap cameras; but my government tells me I shouldn’t trust them, and I [https://www.fcc.gov/document/fcc-bans-authorizations-devices-pose-national-security-threat listen to &amp; believe everything that my government tells me].

I will want to limit its access to the internet, and other machines. Let’s say it connects via wifi.

You can block it from connecting to the internet by its IP - but what if it tries to change its IP? You could create a static mapping in pfSense based on its MAC address, but what if it spoofs its MAC address? If this device were truly malicious, it could do the following:

* Spoof its MAC address to get around a static mapping
* Try to connect using every single IP address
* See if it eventually finds an IP address in that subnet that allows it to go online &amp; connect to other networks/devices
* Upload audio recordings of you saying you had a celebrity crush on Sabrina Carpenter, or that you cry listening to Tori Amos’ Baker Baker. Where’d your reputation be then?

If you want to be more stringent with this - if you genuinely believe your refridgerator is out to get you by recording your intimate moments &amp; blackmailing you with them(it’s probably not), we can make a separate network for them.

We’ll create two separate networks:

* '''Main Network''': <code>192.168.5.0/24</code> for trusted devices ''(we’ve already created this)''
* '''Guest Network''': <code>192.168.7.0/24</code> for untrusted devices ''(needs to be created)''

<blockquote>'''Note''': This is not a normal wifi access point. it is an enterprise level device that allows seamless switching between multiple access points, so that if you have a giant area you never lose your connection or connection strength. The downside is that this isn’t as simple as a standard wifi router, this isn’t your linksys wrt54g from 2005 you configure by typing <code>192.168.1.1</code> and typing in <code>admin</code> for the user &amp; password. You need to install controller software to use it; and it’s worth it. These access points like the eap610 can be found used on ebay in liquidation sales for $45, which is cheaper than a lot of wifi routers.
</blockquote>
Our LAN subnet, where our servers &amp; computers connect to, is <code>192.168.5.0/24</code> meaning that clients connecting here can grab from <code>192.168.5.2</code> to <code>192.168.5.254</code> - <code>192.168.5.1</code> is taken by the router.

Our OpenVPN subnet that we connect to when we use our VPN is <code>192.168.6.0/24</code>meaning that clientst hat connect here can grab from <code>192.168.6.2</code> to <code>192.168.6.254</code> - <code>192.168.6.1</code> is taken by the VPN gateway.

Here we’re going to create <code>192.168.8.0/24</code> as another subnet.

If you’re trusted wifi, you get to connect to the <code>192.168.5.0/24</code> network. If you are connecting to the untrusted wifi, you get to connect to the <code>192.168.7.0/24</code> untrusted network.

When we set up OpenVPN, pfSense created a firewall rule automatically that allowed the VPN subnet of <code>192.168.6.0/24</code> to connect to everything. We will do the opposite for this network. We can create a rule that blocks all traffic TO and FROM the <code>192.168.7.0/24</code> network. Then, we can create specific allow rules for the very specific devices we want it to connect to. If it’s a thermostat, we allow it a connection to &amp; from to <code>192.168.5.4</code>, our home assistant machine. If it is a camera, we allow it a connection to &amp; from <code>192.168.5.2</code>, our frigate machine.

It doesn’t matter if the device spoofs its MAC address to get around a static mapping at this point. It doesn’t matter if it tries to grab every single IP address on the subnet - because NOTHING on <code>192.168.7.0/24</code> is allowed to connect to anything anyway. So, it’s stuck.

This is more “secure” if your threat model includes a thermostat with a hidden microphone in it connected to your wifi, that might want to get around being blocked from phoning home.

# Can’t access your main network
# Can’t see your devices
# Can still access the internet

This is what VLANs are for. We’ll create two completely separate networks:

* Main Network (192.168.5.0/24): For your trusted devices
* Guest Network (192.168.7.0/24): For everyone else

<div class="figure">

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:image-20241113190058398.png
</gallery>

</div>
<div class="figure">

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:image-20241113190156285.png
</gallery>

</div>
<div class="figure">

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:image-20241113190459933.png
</gallery>

</div>
<div class="figure">

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:image-20241113190855998.png
</gallery>

</div>
<div class="figure">

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:image-20241113190925796.png
</gallery>

</div>
<div class="figure">

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:image-20241113190952807.png
</gallery>

</div>
<div class="figure">

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:image-20241113191027630.png
</gallery>

</div>
<div class="figure">

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:image-20241113191045988.png
</gallery>

</div>
<div class="figure">

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:image-20241113191714961.png
</gallery>

</div>
<div class="figure">

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:image-20241113192027212.png
</gallery>

</div>
<div class="figure">

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:image-20241113191736911.png
</gallery>

</div>
<div class="figure">

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:image-20241113191815241.png
</gallery>

</div>
<div class="figure">

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:image-20241113192136101.png
</gallery>

</div>
<div class="figure">

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:image-20241113192201701.png
</gallery>

</div>
<div class="figure">

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:image-20241113192215825.png
</gallery>

</div>
<span id="step-2-pfsense-configuration-guide-for-trusted-untrusted-networks"></span>
== Step 2: PfSense Configuration Guide for Trusted &amp; Untrusted Networks ==

We want to have two separate networks; but we are using one cable to connect the switch to our wifi access point. We do that with VLANs, which are “virtual” LANs. Each packet we send is going to have a tag on it that tells it which LAN it is. The switch, &amp; in the case the wifi access point, will use this to direct the traffic to the correct virtual LAN.

Each of our wifi clients will be connecting to a LAN. The trusted wifi network will connect to the standard <code>192.168.5.0/24</code> LAN, and the untrusted to a 2nd network we create on <code>192.168.7.0/24</code>

<span id="create-vlans"></span>
==== 2.1 Create VLANs ====

# Navigate to: '''Interfaces &gt; Assignments &gt; VLANs'''
# Click “Add” to create first VLAN:
#* Parent Interface: Select your LAN interface (usually igb0 or em0)
#* VLAN Tag: 7
#* Priority: leave blank
#* Description: '''“maliciouswifi”'''
#* Click '''“Save”'''

<span id="create-network-interfaces"></span>
==== 2.2 Create Network Interfaces ====

# Go to: '''Interfaces &gt; Assignments'''
# From the '''“Available network ports”''' dropdown:
#* Select the VLAN 7 interface and click “Add”
#* Note the names assigned (typically OPT1 and OPT2
#* Name this '''maliciouswifi'''

<span id="set-ip-range-of-new-interface"></span>
==== 2.3 Set IP range of new interface ====

# Go to: '''Interfaces &gt; MALICIOUSWIFI'''
# In '''“General Configuration”''' set the following options:
#* Set '''“Description”''' to maliciouswifi
#* Set '''“IPv4” Configuration Type”''' to Static IPv4
#* Set '''“IPv6 Configuration type”''' to None.
#** If you have a reason to use IPv6, you are probably a network administrator for the world trade tower or a mall or something &amp; aren’t reading this guide anyway.
# In '''“Static IPv4 Configuration”''' set the following options:

* '''“IPv4 Address”''' to <code>192.168.7.1</code>
** The slash thingie at the end to <code>/24</code> - this means we get the entire range from <code>192.168.7.2</code> to <code>192.168.7.254</code> for wifi clients connecting to this network when we set up DHCP server.
* Set '''“IPv4 Upstream Gateway”''' to None

<ol start="4" style="list-style-type: decimal;">
<li>Hit '''“Save”'''</li></ol>

<span id="configure-dhcp-server"></span>
==== 2.4 Configure DHCP Server ====

DHCP is what allows you to connect to a wifi network and get online without having to specify the IP address, gateway, DNS server, etc. This is necessary so clients get an IP address when they connect to the wifi network automatically.

* Malicious wifi Network DHCP:

# Navigate to: '''Services &gt; DHCP Server &gt; MALICIOUSWIFI'''

* The interface maliciouswifi will be at the top after you click onto '''“DHCP Server”'''

<ol start="2" style="list-style-type: decimal;">
<li>Configure:
<ul>
<li>Enable: ✓ Checked '''” Enable DHCP server on MALICIOUSWIFI interface “'''</li>
<li>'''“Address Pool Range”''':
<ul>
<li>From: 192.168.7.2</li>
<li>To: 192.168.7.254</li></ul>
</li></ul>
</li>
<li>Click Save</li></ol>

<div class="figure">

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:image-20241113192410257.png
</gallery>

</div>
<div class="figure">

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:image-20241113192442850.png
</gallery>

</div>
<div class="figure">

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:image-20241113192623441.png
</gallery>

</div>
<div class="figure">

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:image-20241113192834456.png
</gallery>

</div>
<div class="figure">

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:image-20241113193208715.png
</gallery>

</div>
<span id="step-3-configure-firewall-rules"></span>
== Step 3: Configure Firewall Rules ==

Now, we’re going to block this from connecting to anything.

<span id="block-maliciouswifi-to-everything"></span>
==== 3.1 Block maliciouswifi to everything ====

<ol style="list-style-type: decimal;">
<li><p>Navigate to: '''Firewall &gt; Rules &gt; MALICIOUSWIFI'''</p></li>
<li><p>Add this rule:</p>
<ol style="list-style-type: lower-alpha;">
<li>Block Inter-VLAN Access:</li></ol>

<ul>
<li>Action: Block</li>
<li>Interface: '''“MALICIOUSWIFI”'''</li>
<li>Protocol: Any</li>
<li>Source: Any</li>
<li>Destination: Any</li>
<li>Description: '''“Block maliciouswifi access to everything”'''</li>
<li>Click Save</li></ul>
</li></ol>

==== 3.2 Add allow rules for devices you wish to speak to one another. ====

Right now devices connected to this wifi network can’t connect to anything. Even if it were a malicious device that were going to try every IP on this subnet after spoofing its MAC address and try to get access to the outside world, it’s stuck.

We would want to add rules '''ABOVE''' the ''“Block maliciouswifi access to everything”'' rule for things we did want to talk.

For instance, let’s say a wireless camera were attached here. We would want to add a rule to allow traffic from the camera, let’s say it’s at <code>192.168.7.15</code> to the frigate machine at <code>192.168.5.2</code> , and then another rule to allow traffic from the frigate machine to the camera. This rule would be listened to ''before'' the rule to block everything.

You can use this to make sure that the thermostat only communicates with home assistant, that the fish camera only communicates with your VPN, etc. It’s a great way to keep untrusted devices from having rampant access to everything.

<span id="step-4-tp-link-omada-controller-sdn-installation-guide"></span>

== Step 4: TP-Link Omada Controller SDN Installation Guide ==

<span id="optional-note-for-the-paranoidskip-ahead-if-not-paranoid"></span>
==== 4.0 Optional note for the paranoid(skip ahead if not paranoid) ====

'''To be clear, if you’re at this level of paranoia, just find a router that has meshing with openwrt and deal with the lower level of performance with switching you’ll get with it. I have yet to find an open source access point + open source firmware that is even close to closed source ones with regards to seamless roaming across multiple access points without dropoffs or slowdowns'''

If you have a problem with running closed source software from a company headquartered in Shenzhen on your computer - I don’t blame you. Rather than install this onto your host system, you can install it onto a virtual machine you do not allow to access the internet, that runs nothing but this software. You would install the virtual machine for omada the same way you would install the virtual machine for mailcow. We have done this many times - simply follow the instructions we’ve already followed, with the following changes:

<ul>
<li><p>When installing Ubuntu server, choose minimal install in the installer.</p></li>
<li><p>Set the IP to 192.168.5.7 instead of 192.168.5.3 we chose for mailcow</p></li>
<li><p>Set the hostname &amp; name of the computer to '''wifitool'''</p></li>
<li><p>Set the static mapping in pfsense with hostname '''wifitool'''</p></li>
<li><p>Make a pfSense firewall rule blocking all traffic '''to''' and '''from''' <code>192.168.5.7</code> on the LAN interface for any protocol, so it looks like this:</p>
<div class="figure">

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:image-20241114175555928.png
</gallery>

</div></li></ul>

<div class="figure">

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:image-20241114175638119.png
</gallery>

</div>
<div class="figure">

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:image-20241114175738722.png
</gallery>

</div>
<div class="figure">

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:image-20241114175818156.png
</gallery>

</div>
Lastly, if you want a level of paranoia that matches [https://www.reuters.com/world/us/us-lawmakers-urge-probe-wifi-router-maker-tp-link-over-fears-chinese-cyber-2024-08-15/ congress], you can set up temporary pfSense firewall rules that block the computer you use to access the tp-link omada controller in your web browser from connecting as well - and toggle them on each time you run the tp-link omada controller software in your browser, and make a rule blocking the IP address of each individual access point from going online as well.

<span id="prepare-the-system"></span>
==== 4.1 Prepare the System ====

Before installation, remove any conflicting packages like older MongoDB versions, Java, or remnants of previous Omada installations to avoid conflicts. '''We never installed these packages onto our server, so they should not be there. Just in case they are. To be clear, you should not have any use for these packages at this point if you’ve been following this guide. '''

<pre>sudo apt purge -y mongodb-org* openjdk-11-* openjdk-8-* jsvc
sudo apt autoremove -y
sudo apt clean</pre>
<span id="install-java-8-and-mongodb"></span>
==== 4.2 Install Java 8 and MongoDB ====

Install Java 8, as the Omada Controller requires it, and install MongoDB (v7.0 is recommended here). It wants old Java. Not version 11.

<pre>sudo apt update

# Some of this software you may already have. No big deal, it doesn't hurt to make sure. 
sudo apt install -y openjdk-8-jre-headless jsvc curl gnupg lsb-release

curl -fsSL https://pgp.mongodb.com/server-7.0.asc | sudo gpg -o /usr/share/keyrings/mongodb-server-7.0.gpg --dearmor
echo &quot;deb [arch=amd64,arm64 signed-by=/usr/share/keyrings/mongodb-server-7.0.gpg] https://repo.mongodb.org/apt/ubuntu $(lsb_release -sc)/mongodb-org/7.0 multiverse&quot; | sudo tee /etc/apt/sources.list.d/mongodb-org-7.0.list

sudo apt update

sudo apt install -y mongodb-org</pre>
<pre>sudo systemctl enable mongod --now
sudo systemctl status mongod</pre>
<blockquote>'''IMPORTANT NOTE:''' mongodb is expecting you to be using an older version of Ubuntu Linux(22.04, codename “jammy”) for this to work. We are using Ubuntu Server (24.04, code name “noble”). There is nothing wrong with this''(besides the fact that I subjected you to ubuntu in the first place, but that’s a conversation for another time)''. 24.04 is the latest stable, long term release. However, mongodb still thinks that jammy is the latest long term/stable release.

If mongodb does not have a repository for ubuntu 24.04 jammy by the time this guide is released, you will have to make the following edit for apt to let you install mongdo from this repository:
</blockquote>
<pre># Open source list file for mongodb for editing
sudo nano -w /etc/apt/sources.list.d/mongodb-org-7.0.list</pre>
<pre># Find the following line:
deb [arch=amd64,arm64 signed-by=/usr/share/keyrings/mongodb-server-7.0.gpg] https://repo.mongodb.org/apt/ubuntu noble/mongodb-org/7.0 multiverse</pre>
<pre># Replace the word `noble` with `jammy`
deb [arch=amd64,arm64 signed-by=/usr/share/keyrings/mongodb-server-7.0.gpg] https://repo.mongodb.org/apt/ubuntu jammy/mongodb-org/7.0 multiverse</pre>
'''The steps in the three grey code boxes above are only necessary if you received an error while trying to install mongodb'''

<span id="find-omada-sdn-controller-software-on-tp-links-website-to-download"></span>
==== 4.3 Find Omada SDN Controller Software on tp-link’s website to download ====

Download the latest .deb package from [https://www.tp-link.com/us/support/download/omada-software-controller/ TP-Link’s Download section]. Right click the download button, click '''copy link''' in your browser, and paste it into the command below:

<pre># Make subdirectory for storing programs if it isn't already there in our home directory

mkdir -p ~/Downloads/programs
cd ~/Downloads/programs

# Check TP-Link's website for the latest version of this sfotware, it should be a .deb file with a filename that looks something like what you see below, just with a newer version

wget https://static.tp-link.com/upload/software/2024/202411/20241101/Omada_SDN_Controller_v5.14.32.3_linux_x64.deb</pre>
<span id="install-the-omada-controller"></span>
==== 4.4 Install the Omada Controller ====

Install the Omada Controller SDN package. If dependencies are flagged, ignore them to proceed with the installation.

<pre>sudo dpkg --ignore-depends=jsvc -i Omada_SDN_Controller.deb
# Just in case anything funny happened while installing an ancient version of java
sudo apt --fix-broken install</pre>
<span id="verify-it-installed-start-the-controller"></span>
==== 4.5 Verify it installed &amp; Start the Controller ====

The Omada Controller should now be running. Access the Omada interface by navigating to <code>https://192.168.5.2:8043</code>.

<blockquote>'''NOTE:''' If it gets stuck on “Starting Omada Controller. Please wait….” and keeps outputting dots, and never starts, and it gives you a bs error about java virtual machine not being available, you followed TP-Link’s documentation instead of mine. Do not pass go, do not collect $200, go directly to jail. That is your punishment for expecting GNU/Linux documentation for a piece of software to work; and you deserve it.
</blockquote>

-----

To enable it on boot, type <code>systemctl enable tpeap</code> , but it should already be starting on boot.

<span id="step-4.5-vlan-tags"></span>
== Step 4.5: VLAN tags ==

<div class="figure">

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:image-20241113114803712.png
</gallery>

</div>
This can be confusing. There are $250 wifi routers that, when put in wifi bridge mode to be used as a switch, will not pass VLAN tags properly. Then there are $20 [https://www.netgear.com/business/wired/switches/unmanaged/gs308/ Netgear GS308v3 switches] that support VLAN tags perfectly.

You don’t have to spend a lot of money to get a switch that has VLAN tags. How do you tell if yours supports VLAN tags? Good question. Netgear’s [https://www.downloads.netgear.com/files/GDC/Unmanaged_Switches/300-Series_Gigabit_DS.pdf datasheet for the GS308] and their [https://www.downloads.netgear.com/files/GDC/GS308v3/GS305v3_GS308v3_IG_EN.pdf instructional manual for the GS308] do not mention the word “VLAN” - not even once. It says it supports 802.1p QOS, but that is not 802.1Q VLAN tagging.

Most modern switches DO support this; but what if you have an old one? What if you are re-purposing an old wifi router as a switch for this setup? Many wifi routers, even older ones, have settings that allow them to be used as a wireless bridge.

As I have said earlier on, when people tell you to ''“RTFM”'', what they are actually saying is ''“eat shit and die”'' - it’s their way of expressing that they hate you. Manuals are functionally useless for 99% of products sold, and rarely if ever answer actual questions. They answer questions that can be answered intuitively without a manual.

<div class="figure">

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:image-20241113115922968.png
</gallery>

</div>
My best answer is as follows; if you are going to have a very small home network, the Netgear GS308 is a great pick that works with VLAN tags. It’s dirt cheap and a workhorse. If you want something that is more upscale, I’d suggest looking at the [https://www.tp-link.com/us/business-networking/omada-switch-l3-l2-managed/sg3218xp-m2/v1/ TP-Link Omada SG3218XP-M2] &amp; other switches in that series, for the following reasons:

'''2.5 GbE speeds'''

Most switches have gigabit ports. This means 1 gigabit - which translates to '''100-120 megabytes per second''' in the real world. Around 2009 when these started to become cheaper(sub-$200), this was more than enough, since hard drives of the time were in the 70-120 megabyte per second range. This meant that it made no sense to pay extra for a switch with more bandwidth, since your hardware was not capable of making use of the extra bandwidth. Whether using a $10,000 switch or the $50 1 gigabit switch, your transfer speed would be the same.

As time has moved on, even cheap desktop hard drives do over 180-250 megabytes per second, and cheapie solid state drives can achieve 200-400 megabyte per second read &amp; write easily. 1 gigabit ports on switches mean you are losing out on transfer speed.

2.5 GbE switches are capable of '''270-290''' megabytes per second,approximately, in the real world. This is still under the capability of more expensive NVME solid state drives, but it is over double what you get with the old gigabit switches.

'''Power over Ethernet(PoE)'''

If you do plan on setting up security cameras, PoE means that you can plug the ethernet cable into the camera without having to run a separate line for power. The power for the camera is provided by the switch through the ethernet cable.

'''Easy management using Omada controller software'''

If you want to have fun with some of this switch’s other features, you can use the same software we’ll be using for EAP-610 wireless access points to control the switch.

<div class="figure">

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:image-20241113193732516.png
</gallery>

</div>
<div class="figure">

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:image-20241113193805590.png
</gallery>

</div>
<div class="figure">

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:image-20241113194116940.png
</gallery>

</div>
<div class="figure">

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:image-20241113194135465.png
</gallery>

</div>
<div class="figure">

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:image-20241113194243719.png
</gallery>

</div>
<div class="figure">

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:image-20241113194341452.png
</gallery>

</div>
<div class="figure">

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:image-20241113194419692.png
</gallery>

</div>
<div class="figure">

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:image-20241113194432870.png
</gallery>

</div>
<div class="figure">

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:image-20241113194509149.png
</gallery>

</div>
<div class="figure">

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:image-20241113194551382.png
</gallery>

</div>
<div class="figure">

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:image-20241113194714308.png
</gallery>

</div>
<div class="figure">

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:image-20241113194736836.png
</gallery>

</div>
<div class="figure">

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:image-20241114081503037.png
</gallery>

</div>
<div class="figure">

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:image-20241114081612226.png
</gallery>

</div>
<span id="step-5-configuring-tp-link-eap610-vlans-in-omada-controller"></span>

== Step 5: Configuring TP-Link EAP610 VLANs in Omada Controller ==

<span id="loading-controller-adopting-your-access-point"></span>
==== 5.1 Loading controller &amp; adopting your access point ====

<ol style="list-style-type: decimal;">
<li><p>Visit Omada Controller in your browser:</p>
<pre>https://192.168.5.2:8043</pre>
<blockquote><p>'''NOTE:''' Take a close look at the IP address &amp; port in the terminal and visit the URL it tells you to upon finishing the installation of TP-Link Omada controller software.</p></blockquote></li>
<li><p>Adopt the access point that matches the IP address you see in pfSense under '''Diagnostics –&gt; ARP Table''' or under</p>
<ul>
<li>Go to Devices</li>
<li>check that EAP610 shows as '''“Connected”'''</li>
<li>If not adopted, use “Adopt” button</li></ul>
</li></ol>

<span id="navigate-to-where-we-create-a-new-network"></span>

==== 5.2 Navigate to where we create a new network ====

# Click on the zone you just created on the main homepage under '''Site List''' once you log in. In our case, that is '''home_demo'''
# Click on '''“Settings”''' in the lower left corner.

* Make sure you clicked on a zone first - if you click on ''“Settings”'' in the lower left corner it will take you to the settings for the controller program rather than for the zone you’re setting up for wifi.

<ol start="3" style="list-style-type: decimal;">
<li><p>Click '''“Wireless Networks”'''</p></li>
<li><p>Click '''“Create New Wireless Network”'''</p></li></ol>

<span id="configure-the-easy-settings-for-the-network"></span>
==== 5.3 Configure the easy settings for the network ====

# Fill in all the usual settings for normal wifi setup you’ve done before on normal wifi routers
#* '''SSID''': maliciouswifi
#** this is the name of the network that shows up when you search for wifi networks on your laptop or phone
#* '''Device type:''' EAP
#** Band: 2.4 GHz, 5 GHz
#* '''Security Key''': whatever password you want for connecting to it
#** This is the wifi password for the network

<span id="configure-vlan-settings-youre-likely-not-familiar-with-if-youre-reading-this"></span>
==== 5.4 Configure VLAN settings you’re likely not familiar with if you’re reading this ====

# Click '''“Advanced Settings”'''
# Set '''“VLAN”''' to '''“Custom”''' and '''“Add VLAN'''” should show up as a new menu item.
# Choose '''“By VLAN ID”''' when the '''“Add VLAN”''' part shows up after you click '''“Custom”'''
# Set the number to 7, which we chose when making the VLA N in pfSense.

<div class="figure">

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:image-20241114111018846.png
</gallery>

</div>
<div class="figure">

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:image-20241114111105623.png
</gallery>

</div>
<div class="figure">

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:image-20241114111257746.png
</gallery>

</div>
<div class="figure">

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:image-20241114111344084.png
</gallery>

</div>
<div class="figure">

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:image-20241114111434791.png
</gallery>

</div>
<div class="figure">

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:image-20241114111555972.png
</gallery>

</div>
<div class="figure">

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:image-20241114111626689.png
</gallery>

</div>
<span id="step-6-make-sure-blocking-rules-work"></span>
== Step 6: Make sure blocking rules work ==

# Connect your phone to this network. Don’t use a VPN. Turn VPN off.
# Try connecting to the web or to home assistant, or anything we set up. It shouldn’t work.
# Add a firewall rule to allow traffic to &amp; from the IP address your phone has grabbed, to the home assistant VM which we set up at <code>192.168.5.4</code>
# Try to access the home assistant VM now on your phone.
# If it works now, but didn’t before, you did a good job.

You can now connect untrusted wifi IoT devices to this and be confident that there is a slightly lower chance that your refridgerator is going to report you fapping back to the manufacturer.

= How to Set Up VLC on Android to Play Videos from a Samba Server =

What’s the point of hoarding 100 terabytes of recipes and GNU/Linux ISOs if you can’t enjoy them on your home entertainment system or your smartphone; no matter where you are in the world?

I’ll start with the phone setup and then move on to the home entertainment system. A lot of people think my GNU/Linux setup that connects to my home entertainment system is way more complicated than it actually is. It’s simpler than you think.

<div class="figure">

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:vlc001070a_hometheeater_ANDROIDCAP_screen-20241116-140426.png
</gallery>

</div>
<div class="figure">

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:vlc002070a_hometheeater_ANDROIDCAP_screen-20241116-140426.png
</gallery>

</div>
<div class="figure">

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:vlc004070a_hometheeater_ANDROIDCAP_screen-20241116-140426.png
</gallery>

</div>
<div class="figure">

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:vlc006070a_hometheeater_ANDROIDCAP_screen-20241116-140426.png
</gallery>

</div>
<div class="figure">

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:vlc007070a_hometheeater_ANDROIDCAP_screen-20241116-140426.png
</gallery>

</div>
<div class="figure">

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:vlc008070a_hometheeater_ANDROIDCAP_screen-20241116-140426.png
</gallery>

</div>
<div class="figure">

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:vlc009070a_hometheeater_ANDROIDCAP_screen-20241116-140426.png
</gallery>

</div>
<div class="figure">

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:vlc010070a_hometheeater_ANDROIDCAP_screen-20241116-140426.png
</gallery>

</div>
<div class="figure">

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:vlc012070a_hometheeater_ANDROIDCAP_screen-20241116-140426.png
</gallery>

</div>
<div class="figure">

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:vlc013070a_hometheeater_ANDROIDCAP_screen-20241116-140426.png
</gallery>

</div>
<div class="figure">

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:vlc019070a_hometheeater_ANDROIDCAP_screen-20241116-140426.png
</gallery>

</div>
<div class="figure">

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:vlc020070a_hometheeater_ANDROIDCAP_screen-20241116-140426.png
</gallery>

</div>
<div class="figure">

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:vlc021070a_hometheeater_ANDROIDCAP_screen-20241116-140426.png
</gallery>

</div>
<div class="figure">

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:vlc022070a_hometheeater_ANDROIDCAP_screen-20241116-140426.png
</gallery>

</div>
<div class="figure">

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:vlc025070a_hometheeater_ANDROIDCAP_screen-20241116-140426.png
</gallery>

</div>
<div class="figure">

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:vlc026070a_hometheeater_ANDROIDCAP_screen-20241116-140426.png
</gallery>

</div>
<div class="figure">

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:vlc029070a_hometheeater_ANDROIDCAP_screen-20241116-140426.png
</gallery>

</div>
<div class="figure">

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:vlc030070a_hometheeater_ANDROIDCAP_screen-20241116-140426.png
</gallery>

</div>
<div class="figure">

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:vlc036070a_hometheeater_ANDROIDCAP_screen-20241116-140426.png
</gallery>

</div>
<div class="figure">

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:vlc040070a_hometheeater_ANDROIDCAP_screen-20241116-140426.png
</gallery>

</div>
<div class="figure">

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:vlc042070a_hometheeater_ANDROIDCAP_screen-20241116-140426.png
</gallery>

</div>
<div class="figure">

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:vlc044070a_hometheeater_ANDROIDCAP_screen-20241116-140426.png
</gallery>

</div>
<span id="step-1-installing-vlc-on-android"></span>

== Step 1: Installing VLC on Android ==

<span id="download-vlc-from-the-f-droid-store"></span>
==== 1.1 Download VLC from the F-Droid Store ====

# Go to f-droid store app
# Search for VLC
# Download VLC
# Install VLC
# be happy

<div class="figure">

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:vlc006070a_hometheeater_ANDROIDCAP_screen-20241116-140426.png
</gallery>

</div>
<div class="figure">

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:vlc007070a_hometheeater_ANDROIDCAP_screen-20241116-140426.png
</gallery>

</div>
<div class="figure">

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:vlc008070a_hometheeater_ANDROIDCAP_screen-20241116-140426.png
</gallery>

</div>
<div class="figure">

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:vlc009070a_hometheeater_ANDROIDCAP_screen-20241116-140426.png
</gallery>

</div>
<div class="figure">

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:vlc010070a_hometheeater_ANDROIDCAP_screen-20241116-140426.png
</gallery>

</div>
<div class="figure">

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:vlc012070a_hometheeater_ANDROIDCAP_screen-20241116-140426.png
</gallery>

</div>
<div class="figure">

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:vlc013070a_hometheeater_ANDROIDCAP_screen-20241116-140426.png
</gallery>

</div>
<div class="figure">

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:vlc019070a_hometheeater_ANDROIDCAP_screen-20241116-140426.png
</gallery>

</div>
<div class="figure">

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:vlc020070a_hometheeater_ANDROIDCAP_screen-20241116-140426.png
</gallery>

</div>
<span id="step-2-adding-samba-share-as-a-favorite"></span>
== Step 2: Adding Samba share as a “favorite” ==

# '''Open VLC''': Once you have it installed, open it up.
# Grant VLC the permissions it asks for, if you want it to find files on your phone &amp; be able to play them.
# '''Add a Server''':
#* Go to <code>Browse</code>.
#* Click on the three dots in the upper right corner.
#* Select <code>Add a server favorite</code>.
# '''Choose Protocol''':
#* Select <code>SMB</code> as your protocol. It defaults to FTP, but we want SMB.
# '''Enter Server Address''':
#* Type in the server address where your ZFS pool is located. This could be something like <code>happycloud.com</code> or simply the IP address: <code>192.168.5.2</code>.
# '''Server Name''':
#* Enter a server name, like <code>ZFS pool</code>, and hit OK.

<div class="figure">

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:vlc021070a_hometheeater_ANDROIDCAP_screen-20241116-140426.png
</gallery>

</div>
<div class="figure">

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:vlc022070a_hometheeater_ANDROIDCAP_screen-20241116-140426.png
</gallery>

</div>
<div class="figure">

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:vlc025070a_hometheeater_ANDROIDCAP_screen-20241116-140426.png
</gallery>

</div>
<div class="figure">

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:vlc026070a_hometheeater_ANDROIDCAP_screen-20241116-140426.png
</gallery>

</div>
<div class="figure">

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:vlc029070a_hometheeater_ANDROIDCAP_screen-20241116-140426.png
</gallery>

</div>
<div class="figure">

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:vlc030070a_hometheeater_ANDROIDCAP_screen-20241116-140426.png
</gallery>

</div>
<div class="figure">

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:vlc036070a_hometheeater_ANDROIDCAP_screen-20241116-140426.png
</gallery>

</div>
<div class="figure">

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:vlc040070a_hometheeater_ANDROIDCAP_screen-20241116-140426.png
</gallery>

</div>
<div class="figure">

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:vlc042070a_hometheeater_ANDROIDCAP_screen-20241116-140426.png
</gallery>

</div>
<div class="figure">

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:vlc044070a_hometheeater_ANDROIDCAP_screen-20241116-140426.png
</gallery>

</div>
<span id="step-3-find-your-hidden-share"></span>
== Step 3: Find your hidden share ==

Once you’ve added the server to your favorites, you might notice nothing pops up on the screen. Plus, you might see items in favorites you never added. Don’t worry, it’s just open source quirks. Scroll over, and you’ll see the share you added. No mistakes here—it’s just open source being open source.

**Make sure you’re connected to your VPN!

Before you can connect to your share, make sure you’re attached to your VPN. Without it, you won’t be able to access your share. Once connected, you can click on your share!

# '''Authenticate''':
#* You’ll see <code>Archive</code>, but you need to authenticate with your username and password.
#* Enter your credentials. You can save them in VLC or use a password manager like Bitwarden—whatever floats your boat. I’ll save it in VLC for now.
# '''Access Files''':
#* Click on your file, and let’s watch. Ignore any video player tips. And there you have it! That’s how you connect to your share from anywhere in the world to view your files.

<span id="alternative-programs"></span>
== Alternative Programs ==

There are also non-open source programs that are pretty good at browsing, like '''Owl Files'''. You’d set them up similarly by entering your Samba credentials and network share information. Connect to your VPN, and your files are right there and available for you.

And that’s it! Enjoy your media from your ZFS pool wherever you are.

<span id="setting-up-a-linux-based-home-entertainment-system"></span>
= Setting Up a GNU/Linux-Based Home Entertainment System =

Here we’ll set up a living room stereo &amp; source for your television for fun. Welcome to the portion of our guide where we dive into setting up a GNU/Linux computer and a Samba server as the heart of your living room entertainment center. I’ll also walk you through setting up a hi-fi stereo system to achieve high-fidelity sound affordably, with no audiophile snake oil.

<div class="figure">

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:image-20241116221000894.png
</gallery>

</div>
<div class="figure">

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:image-20241116215926331.png
</gallery>

</div>
<div class="figure">

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:image-20241116220449259.png
</gallery>

</div>
<div class="figure">

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:image-20241116220837576.png
</gallery>

</div>
<div class="figure">

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:image-20241116221436445.png
</gallery>

</div>
<span id="hooking-everything-up"></span>
== Hooking everything up: ==

<span id="step-1-connect-your-computer-to-your-televison-with-an-hdmi-cable."></span>
=== Step 1: Connect your computer to your televison with an HDMI cable. ===

I hope I don’t have to explain this one. I wouldn’t use your server for this, I use a computer I have in my living room.

<span id="you-dont-need-a-powerful-computer-for-an-entertainment-system"></span>
==== 1.1 You don’t need a powerful computer for an entertainment system ====

Most modern CPUs (Intel/AMD) have integrated graphics that support hardware-accelerated video decoding for formats like H.264, HEVC (H.265), and VP9. This means:

* Nearly anything made in the past 10 years, even cheap stuff, will be fine for most video playback.
* You do not need to buy a GPU to watch stuff in your living room.

If buying a dedicated living room computer, II would suggest buying a machine that has optical audio output so that you can output digital audio to a receiver or DAC(digital to analog converter); rather than doing the audio processing on the computer, this sends the digital signal to another device to do it. If your motherboard’s analog audio output/headphone jack is noisy and you don’t have a digital output, you’re stuck paying for an audio interface to get good sound or digital output.

It is difficult to recommend cheap pre-built PCs as many have loud fans due to poor cooling that become annoying in an environment where you will listen to music and quiet scenes in youtube videos, movies &amp; television. If you’re reading this guide, you most likely will want to build one yourself.

<span id="understanding-hdmi-cable-requirements"></span>
==== 1.2 Understanding HDMI Cable Requirements ====

If your computer is far away from the television, and you want to do 4k, you may want a 50 ft cable. The problem is that most 50 ft cables that advertise they do 4k are a scam. The vendors prey upon people that can’t tell the difference between 4k30hz(30 frames per second) and 4k60(4k resolution at 60 frames per second).

Here’s what you need to know:

* Integrated graphics is fine - you do not need a dedicated GPU to playback 4k 60 hz video content. A decent CPU made sometime in the past 10 years is more than enough.
* Expensive HDMI cables will never make a difference in picture quality. They either do 4k at 60 hz or they don’t.
* HDMI 2.0 bandwidth requirements:
** 4K @ 30Hz requires about 8.16 Gbps
** 4K @ 60Hz requires about 16.32 Gbps
** Any cable claiming “18 Gbps” should handle 4K60 if it actually meets specs. Most amazon/walmart no-name junk don’t.

<span id="video-cable-buying-guide"></span>
==== 1.3 Video Cable Buying Guide ====

* A [https://www.monoprice.com/product?p_id=15429 cheap sub-$10 cable from monoprice] is fine for shorter runs(25 feet or less) and will do 4k60 with ease.
* If impatient, you can buy a [https://www.walmart.com/ip/GE-15ft-HDMI-2-0-Cable-with-Ethernet-Gold-Plated-Connectors-48722/ high quality, General Electric HDMI cable that does 4k60 from walmart] for about $20, in store, same day.
* If running more than 25 ft, fiber optic active HDMI cables from reputable vendors like [https://www.monoprice.com/product?p_id=43331 monoprice] or [https://www.bluejeanscable.com/store/hdmi-cables/hdmi-cable.htm bluejeanscable] become necessary because:
** Traditional copper cables have signal degradation over longer distances
** This doesn’t mean a worse picture; rather, you can’t use higher resolutions or framerates.
** Active fiber cables regenerate the signal and pass signal with less degredation allowing 4k at 60 hz(60 fps) over long distances.
** They’re more expensive but actually work at advertised specs unlike amazon/walmart scams.
* There are cheaper 50 ft options, but they are [https://www.walmart.com/ip/HDMI-Cable-4K-Ultra-HD-50-Foot-2-Pack-2-0-Cable-High-Speed-18Gbps-4K-60Hz-HDR-3D-2160p-1080p-HDCP-2-2-ARC-Cables-Monitors-HDTV/564196472 scams].
* bluejeanscable [https://www.bluejeanscable.com/legal/mcp/index.htm humiliated monster.com’s legal department], are honest &amp; upfront about what they sell, produce quality products, and debunk bs on their blog. For this reason alone, they’ve earned my loyalty.

<div class="figure">

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:image-20241116215551750.png
</gallery>

</div>
<div class="figure">

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:image-20241116222626732.png
</gallery>

</div>
<div class="figure">

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:image-20241116231037348.png
</gallery>

</div>
<div class="figure">

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:image-20241116231235888.png
</gallery>

</div>
<div class="figure">

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:image-20241116231341624.png
</gallery>

</div>
<span id="step-2-hook-up-your-computers-sound-output-to-your-stereo."></span>
=== Step 2: Hook up your computer’s sound output to your stereo. ===

<span id="analog-out-from-your-desktop-motherboard-or-laptop-headphone-jack."></span>
==== 2.1 Analog out from your desktop motherboard or laptop headphone jack. ====

A cable like a [https://www.monoprice.com/product?p_id=665 1/8” to stereo RCA] from monoprice allows you to hook up the headphone jack from your computer to many stereo amplifiers and home audio receivers. In my setup, I do not have a receiver that is capable of video - I have a [https://www.reddit.com/r/audiophile/comments/eqpns1/rotel_rb1090_added_to_my_system_this_evening/ 30 year old Rotel RB-1090 tank] with RCA input, so this is what I would use to hook up my laptop or a desktop to my stereo if I didn’t have a separate audio interface.

If you don’t wait to wait for an order, you can also [https://www.walmart.com/ip/onn-6-AUX-to-RCA/1342171538 buy these at your local walmart].

'''Why this will suck''':

The analog audio output from your motherboard is often horrible because you have so much else going on in there. Your GPU, CPU, RAM, are all high bandwidth devices, you have everything on a single circuit board. Things have improved vastly in this regard since I was young and dealt with the horrors of trash like the ac97, where there was audible hissing &amp; warbling that changed in pitch &amp; intensity when you dragged windows around the screen, and weird high frequency sounds depending on the sensitivity of your stereo system. However, it is often still there.

<span id="hdmi-output-from-your-computer-to-your-tv."></span>
==== 2.2 HDMI output from your computer to your TV. ====

If you use the speakers built into your television, you are missing out bigtime. However, there are cases where you have no choice. In this case, the audio and video will get to your television over the HDMI cable, and your setup will be simple.

'''Why this will suck''':

Television speakers are trash.
* They will be filled with cabinet resonances from the giant television.
* You can’t fit speakers that will do a proper job inside of a very thin television.
* The proper location of speakers in your room will not be the same location as the television in your room.

<span id="hdmi-output-from-your-computer-to-your-receiver."></span>
==== 2.3 HDMI output from your computer to your receiver. ====

You may have a setup where you have a receiver that you hook up between your devices &amp; your sound system/TV - in that case, just plug the HDMI out from the computer into that. Then you can use HDMI to carry the sound &amp; the video. This is common in home theater setups where you might a bluray player, a cable box/FIOS TV box, and a game console that plug into a receiver. This receiver usually feeds your television a video feed, and connects directly to your speakers and subwoofer.

'''Why this is better''': You are not using the analog audio output from your laptop or desktop. This allows you the flexibility to choose an audio device that does not have poor sound quality, rather than being stuck with what comes in your computer. Digital output means even if the motherboard’s audio circuit is total garbage, it doesn’t make a difference, since you aren’t using it. You will be sending the raw 1s &amp; 0s of the audio to another device &amp; letting it do the work of turning it into an audio signal.

<span id="optical-output-from-your-computer-to-your-receiver."></span>
==== 2.4 Optical output from your computer to your receiver. ====

Optical audio output is available on most desktop motherboards. It is worth checking to see if yours has this; which is green in the photo above. Most sound cards also have this port, if you have a sound card.

This requires an optical cable, but optical audio cables are considerably cheaper than optical video cables, since the bandwidth requirements are so much lower. For same day purchases, a [https://www.walmart.com/ip/onn-6-Digital-Optical-Audio-Toslink-Sound-Bar-TV-Cable-Black/ cheap walmart optical cable] will do fine, and [https://www.monoprice.com/product?p_id=1419 high quality 50 ft cables] from reputable vendors like monoprice cost less than $15.

'''Why this is better''': You are not using the analog audio output from your laptop or desktop. This allows you the flexibility to choose an audio device that does not have poor sound quality, rather than being stuck with what comes in your computer. Digital output means even if the motherboard’s audio circuit is total garbage, it doesn’t make a difference, since you aren’t using it. You will be sending the raw 1s &amp; 0s of the audio to another device &amp; letting it do the work of turning it into an audio signal.

<blockquote>'''“UHM, AKSHUALLY” NOTICE:''' Some wiseguy’s going to say that this is unnecessary. They might say that if we’re connecting our computer to the television using an HDMI cable, that the audio is ''already'' going to the television through the HDMI cable, and that you can use the optical output from the television to send the audio to the receiver digitally without having to worry about whether your computer has a digital SPDIF output.

Some televisions do this. Some don’t. Some claim they do and have broken menus. If we are going to be buying gear from scratch, I think it makes sense to keep our options open. '''In 2024, there is no price premium to pay by asking for an audio jack that came out in 1983.'''
</blockquote>
<span id="basic-purchase-considerations"></span>
==== 2.5 Basic purchase considerations ====

We will go into this in greater detail later: for now, let’s go over the basics.

If your computer is far away from the receiver or amplifier, you should really consider using an optical cable to connect your computer to the audio source to avoid hiss, distortion, hum, and horrible audio. Even high quality analog audio cables suck when they are unbalanced over long distances; no laptop or standard desktop computer uses balanced output. However, even the cheapest of spdif optical cables will be fine even at 50 ft with audio signals. Digital audio signals are far lower in bandwidth so there is no real worry about degradation at any practical household length of cable.

On the low end, I would suggest a used stereo receiver that has optical audio input from a reputable brand. These can be found on [https://www.ebay.com/sch/i.html?_fsrp=1&_from=R40&Audio%2520Inputs=Digital%2520Optical%2520TOSLINK&Number%2520of%2520Channels=2%7C2%252E1&_nkw=stereo+receiver&_sacat=0&Audio%2520Outputs=Digital%2520Optical%2520TOSLINK%7CDigital%2520Coaxial%2520RCA&LH_BIN=1&rt=nc&Type=Stereo%2520Receiver%7CIntegrated%2520Amplifier&_dcat=14981 eBay by searching with the following filters]:

* '''Number of channels''':
** '''2.1''' if you want to attach a subwoofer later
** '''2''' if you don’t care.
* '''Type''': '''Stereo Receiver''' and '''Integrated Amplifier'''
** This will provide you with volume control, ability to utilize multiple sources, and an amplifier for your speakers.
** I don’t bother with surround sound.
*** Two good speakers will always beat 6 crappy speakers.
*** For any given price point, you get a higher budget when buying two speakers than you do buying six.
* '''Audio Inputs''': '''Digital Optical TOSLINK'''.
** This means the receiver has a '''DAC''' ''(digital to analog converter)''. This turns the 1s &amp; 0s that make up the audio files on your computer into an analog signal.
** The one in your receiver usually does a better job. I’m not talking about audiophile nonsense, just basic competence.
** On lower quality PC motherboards you can literally hear hiss that changes when you do CPU intensive things on your machine when using sensitive stereo equipment or headphones.
* '''Buying Format''': Buy it now.
** Bidding wars on eBay are fun &amp; dopamine releasing. They’re exciting. When people win, they feel like they’ve won until they realize they’ve paid 30% over the used-market-value of what they purchased.
** If buying from auctions on eBay, consider using an '''auction sniper''' like [https://www.gixen.com/main/index.php Gixen.com]. Auction snipers are programs where you input what you wish to pay for something, and it submits your bid seconds before the auction ends. This way, you don’t get caught up in a bidding war. If you bid too early, it allows others the chance to increase their bid as well, which drives up the price.
** Using an auction sniper removes the emotional aspect from auctions that drives up prices, and encourages price-discipline in setting the max you are willing to pay for the item ''early on.''

<span id="step-3-access-media-on-your-samba-share"></span>

=== Step 3: Access Media On Your Samba Share ===

For a media computer, my setup is actually dirt simple. It goes like this.

# Connect computer to TV/stereo.
# Find file I want to play(music or video) in my file explorer(thunar, windows explorer, etc)
# Double click to open in VLC
# Enjoy

Here’s how you can access your files:

# '''File Explorer''': I use <code>Thunar</code> as my file explorer. If you’re on Windows, you’re likely using Windows Explorer.
# '''Accessing Samba Share''':
#* In Windows, you’d type <code>\\</code> followed by the IP address of your ZFS pool computer. For instance, in our case, it would be <code>\\192.168.5.2</code>
#* On Linux, type <code>smb://</code> followed by the IP address or hostname of your share in '''Thunar''' file manager Enter your username and password when prompted. For instance, in our case, it would be <code>smb://192.168.5.2</code><gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:image-20241116214553212.png
</gallery>

Once you do this, you’ll have access to all your files stored on the ZFS pool.

<span id="step-4-play-media-with-vlc"></span>
=== Step 4: Play Media with VLC ===

To enjoy your GNU/Linux ISOs &amp; recipes, simply find your files in the Samba share, double-click to open them in '''VLC'''. Boom, you’re set. [https://www.videolan.org/vlc/ VLC is an open source media player] that is fast, efficient, and supports nearly every audio format, video format, codec etc. on earth. It works on Windows, Mac, GNU/Linux, Android, iPhones, ChromeOS, FreeBSD.. just about everything. .

<span id="putting-together-affordable-home-hi-fi"></span>
= Putting together affordable home hi-fi =

Now, let’s break down the audio components because, you don’t need to spend a fortune for good sound. You just have to avoid snake oil and sound bars.

<span id="speakers"></span>
== Speakers ==

* '''Speakers''': I use <code>Vandersteen Audio</code> speakers. They have minimal cabinet resonance, phase coherent crossovers &amp; driver positioning, and even frequency response for clear sound.
* '''Amplifier''': An Rotel RB-1090 powers my speakers.
* '''Subwoofers''': Two HSU Research ULS-15 from Dr. Hsu, one of the inventors of the original subwoofer.

<div class="figure">

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:vandersteen-2c-loudspeakers-left-on-curb.jpg
</gallery>

</div>
<div class="figure">

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:image-20241116234208414.png
</gallery>

</div>
<div class="figure">

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:image-20241116234642727.png
</gallery>

</div>
<div class="figure">

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:image-20241116234527754.png
</gallery>

</div>
<div class="figure">

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:image-20241116234922873.png
</gallery>

</div>
<span id="speaker-selection-why-i-use-vandersteens"></span>
=== Speaker Selection: Why I Use Vandersteens ===

<span id="minimal-cabinet-resonances-diffraction-off-the-bezel."></span>
==== Minimal cabinet resonances &amp; diffraction off the bezel. ====

These speakers cost $1100 used when I got them, and under $900 were available with minor crossover issues. I like these for a very good reason; exceptional engineering with little/no attention put to marketing or looks. True function over form.

They have great frequency response AND phase response. Further, their shape avoids baffle diffraction &amp; cabinet resonances.

Do this - put your hands by your mouth and cup them. That weird boxy sound you get? That’s what it’s like when you have a speaker that’s a giant box. It’s why your television sounds like garbage.

When you look at these Vandersteens, you notice that even though it looks like a big speaker, the top part is actually just a pole. It’s nothing in there - it’s almost completely hollow besides the bass cabinet. Minimal baffle, minimal diffraction. When there’s diffraction that means you’re listening to the noise from the speaker driver PLUS the reflections off the cabinet that are milliseconds apart.

When you get used to hearing speakers that have minimal cabinet resonances and baffle diffraction, it’s really hard to go back to speakers that do. Everything else sounds like a speaker; this sounds real.

<div class="figure">

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:image-20241116235548018.png
</gallery>

</div>
<div class="figure">

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:image-20241116235521916.png
</gallery>

</div>
<div class="figure">

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:image-20241116235356286.png
</gallery>

</div>
<span id="used-market-availability."></span>
==== Used market availability. ====

'''Inflation? What inflation?'''

In 2009 I bought a set of model 2 for $400 from someone with a leaky apartment, and in 2011 I had the choice of $1100 for high quality used or $900 for a set with minor, repairable crossover issues. Now, a set of 2c are $650, and 3a signatures are $1600 in good condition.

'''Low end models feature same high end engineering'''

99% of what you get in the Vandersteen Model 3a signature which is $7000 new and $1000-$2000 used you get in the lower end models like the 2c.

'''These are always for sale.'''

Anytime you to go ebay.com or audiogon.com , someone is selling a set of these.

'''Subwoofer? Why?'''

These are down about 1 dB at 30 hz. That’s insane. Most likely, the subwoofer you have with your soundbar or home theater produces less bass than these.

The extension along with a small cabinet does come at a price - you’re not going to 120 dB with a lot of low end with these. But, for most music and even movies at reasonable volumes in average sized spaces, you’ll get the full range experience without feeling like you’re missing out.

<span id="speaker-selection-why-i-use-axiom-m3"></span>
=== Speaker Selection: Why I Use Axiom M3 ===

'''Quality engineering over marketing wankery'''

'''Axiom''' is a company that was early to the scene with direct-to-the-consumer online sales. Their “marketing budget” was a guy named Alan Lofft who answered people’s questions on an early webforum that looked like a usenet newsgroup with their logo in the top.

Axiom conducted research at the '''National Research Council''' in Canada, where double-blind tests were performed in which ordinary people would say what they preferred with regards to audio quality. Taking this scientific approach with input from the public, combined with extensive testing and design in a top quality facility allowed them to draw direct conclusions from how speakers measured to what people wanted.

'''Affordability'''

Although their prices have went up for brand new speakers, they can still be found dirt cheap used. Back in the day, their m22 speaker sounded similar to paradigm studio speakers that were near triple the price.

Speakers like the M60 that cost $800 new are no longer competitive deals at their current pricing of $2000/pair - but speakers like the M3 can be found for $160-$220, fit on a desk, and offer exceptional sound for dirt cheap.

'''Exceptional frequency response'''

Speakers like the M3 have a neutral frequency response and a very natural sound.

'''Minimal cabinet resonances'''

Take a look at the M3. Notice how the walls are not parallel? This lessens the type of internal standing waves that occur when a speaker is a perfect cube box. It’s a small touch, but little details like this show them actually focusing on engineering rather than making it look pretty, paying for annoying influencer marketing campaigns, and trendy nonsense.

Same deal with Vandersteens - you can grab the Model 2s used for like $600. These go down to about 30 hertz, very linearly. So you could easily use these without a subwoofer and get better bass than 99% of those computer speaker setups with their tiny subwoofers. These actually have a 10-inch passive radiator in the back and an 8-inch woofer in the front.

The key is don’t buy this stuff new. Just look through eBay for a few minutes, check AudioGon, and you can find insane deals. You’ll end up with speakers that absolutely destroy setups that cost 5-10 times more.

<span id="debunking-audiophile-myths"></span>
== Debunking Audiophile Myths ==

Now, let’s address some audiophile myths. There’s this idea that more expensive always equals better, especially when it comes to cables. You’re going to hear about people justifying $5,000 cables, which is ''absolute nonsense''.

<span id="abx-double-blind-testing-doesnt-matter"></span>
==== ABX Double blind testing doesn’t matter ====

A key sign you’re speaking to someone who has their head as far up their ass as their ego, or a salesman, is when they refuse to acknowledge the benefits of ABX double blind tests. Hydrogenaudio is '''THE''' place for top tier codec developers &amp; programmers to congregate and showcase their new developments; they have had ABX testing as part of their forum rules for over 20 years. If you post about sonic differences without sharing ABX test results.. you’re gone.

That should tell you something.

The ABX test is a method used to objectively compare audio equipment. It involves three inputs—A, B, and X, where X is randomly selected from either A or B. The listener must identify X without prior knowledge, and if they can’t consistently tell the difference, their input is considered irrelevant.

You have a program where you know what A is(an uncompressed wav file), you know what B is(a compressed 128 kbps AAC file), but you don’t know what X is. Every time you hit the X button, you are listening to either A or B - but you don’t know which. It is your job to figure out what X is, each time.

If you can’t get it right '''''12 out of 16 times''''', you didn’t hear a difference. It was all in your head.

Our memory for people’s voices is exceptional. Our ability to be honest with ourselves about our auditory memory is '''complete garbage'''. The reason for this is that we forget what something sounded like with regards to every sonic detail the moment we stop listening to it. It’s easy for our brain to ''“think”'' it heard a difference when it didn’t.

'''This is a good thing, right?'''

It depends how you see it.

<span id="the-upside-of-abx-testing"></span>
===== The upside of ABX testing: =====

If you can’t hear the difference between a $200 amplifier and a $20,000 amplifier, you just avoided breaking into your 401k for $19,800 worth of audiophile bs.

<span id="the-downside-of-abx-testing"></span>
===== The downside of ABX testing: =====

The crushing of fragile egos.

Read youtube comments sections. Anytime a company, a manufacturer, a developer, etc. screws a group of people over, there are two groups of people in the comments:

* People who are supportive
* People who say that everyone who made a difference choice than them in
** Who they voted for
** Who they worked for
** Who they did work for
** What software they bought
** What hardware they bought
** etc, etc. is an idiot.

This is done because it ''makes people feel better about themselves''. If I can hear the difference between a $5 cable and a $500 cable, it means I’m a connoisseur, unlike the plebs &amp; unwashed masses who can’t tell the difference. It also makes me '''feel''' like I am getting an upgrade when I am actually not. Above all, it gives people an ego boost, and who doesn’t want that?

* '''Audio Memory and Bias:'''
** Human auditory memory is fleeting. The moment you switch from one system to another, your ability to accurately remember the sound diminishes.
** This makes it easy to convince yourself that a more expensive component sounds better, even if it doesn’t.

<blockquote>'''Warning:''' Avoid falling for marketing gimmicks that promise crazy improvements whose vendors will hide from ABX testing. Forums like <code>hydrogenaudio</code> can offer a reality check with evidence-based discussions.
</blockquote>
By focusing on what ACTUALLY matters; speaker quality, room acoustics, and well-researched purchases—you can make a hi-fi system that satisfies both your ears and your wallet.

<span id="expensive-equipment-is-a-priority-over-acoustic-treatment"></span>
=== Expensive equipment is a priority over acoustic treatment ===

When you look on audiophile webforums, you will see people with Krell amplifiers, wilson watt puppy speakers, and lavry digital to analog converters in untreated drywall rooms. No bass traps. No acoustic panels. It’s insane.

A $400 stereo in a good room will beat a $40,000 stereo in an untreated room.

Avoid falling into the trap of spending thousands on equipment that doesn’t deliver proportionately better sound. Focus on well-designed, affordable electronics, and you’ll have a setup that works amazing in your living room without emptying your wallet.

<span id="receivers-amps-electronics"></span>
== Receivers, amps, electronics ==

Today, audio electronics that are competently designed(key word; ''competently'') will be indistinguishable in an ABX test from gear that costs $10,000. Paying $10,000 for an amplifier or $8000 for a DAC isn’t an exercise in audible improvements; they’re just status symbols.

<span id="the-basic-building-blocks"></span>
=== The Basic Building Blocks ===

You need three main things to get from digital music to sound: 1. Something to turn digital into analog (DAC) 2. Something to control volume and inputs (preamp) 3. Something to make it loud enough for speakers (power amp)

<span id="digital-to-analog-converter-dac"></span>
==== Digital to Analog Converter (DAC) ====

* Takes the digital signal from your computer, the 1s and 0s, and turns it into an electrical audio signal.
* This could be in your computer motherboard, a soundcard, in a box by itself, in your receiver, or your television.
* Most modern ones are fine - don’t fall for a $10000 DAC or similar bs
* Having this OUTSIDE of your computer usually means less chances for computer-y noise in your audio like hiss/high frequency noises when doing something with your computer.

<span id="preamp"></span>
==== Preamp ====

* This controls which input you’re listening to, so you can switch between a bluray player, cable box, playstation, etc.
* This is what has a volume knob, so it’s pretty much a fancy switch &amp; volume control, sometimes has bass/treble controls on it.
* This isn’t what makes things ''louder'' - that’s for the power amplifier.
* This can be better than taking the output from a digital to analog converter and simply lowering the audio volume in VLC.

<span id="preamp-why-the-hell-would-i-pay-for-a-fancy-volume-knob-when-my-mouse-wheel-and-vlc-let-me-do-that-for-free"></span>
==== Preamp? Why the hell would I pay for a fancy volume knob when my mouse wheel and VLC let me do that for free? ====

When you lower the audio signal volume in VLC, you’re not attenuating an analog signal. Attenuating an analog signal takes the same audio you had and just shrinks the waveform.

When you lower the volume using VLC and then amplifying that, you’re lowering the volume digitally. 16 bit audio has 96 dB of dynamic range, 24 bit audio has 144. If you lower the volume digitally too much, it will start to sound like you are actually losing bits of audio. Even digital preamps usually use a ''digital signal'' to control '''analog amplification &amp; attenuation.'''

A great example of this would be to hook a digital to analog converter with no volume control/attenuation knob up to your computer, and plug it '''straight''' into a power amp. This would be full volume all the time. Then, lower the volume in VLC. This will sound different than lowering the volume on an analog preamp, because you’re not lowering the signal, you’re throwing away digital data.

'''If you think paying extra for a preamp is stupid, just buy a stereo receiver. You get a good enough preamp along with a good enough DAC &amp; amp and it costs around the same or less than just a preamp from “audiophile” brands.'''

<span id="power-amp"></span>
==== Power Amp ====

* Turns a tiny audio signal into a big audio signal.
* As long as it’s competently designed and can power a 4 ohm load without turning off, you’re fine.

<span id="now-the-combinations"></span>
=== Now the Combinations: ===

Similar to how a modern wireless router is actually a router, a switch, and a wireless access point all in one, the devices below are usually combinations of the devices above:

<span id="integrated-amp"></span>
==== Integrated Amp ====

* Preamp + Power amp in one box
* Usually does not come with a digital to analog converter, so still needs a DAC if you have a source with a digital output.
* Usually cheaper than separate components when comparing with others from the same company.

<span id="receiver"></span>
==== Receiver ====

* DAC + Preamp + Power amp all in one
* Often has digital inputs like optical/HDMI
* Has radio tuner, (that is why it’s called a receiver)
* Usually the cheapest all-in-one option
* Good for most people who just want things to work &amp; sound good without being overly complicated.

Most modern electronics that are competently designed all sound basically the same. Don’t fall for that “magical preamp” or “warm sounding DAC” garbage. Get something with enough power for your speakers, digital inputs you need, and spend the rest of your money on speakers and room treatment.

The only time you need separate components is: * Need more power than receivers offer * Want to upgrade one piece at a time * Have some specific feature need * Found a crazy deal on used gear

For most of the people reading this, a used ten year old receiver with optical input will do everything you need, and cost under $200. '''Save your money for the stuff that actually matters; good speakers &amp; acoustic treatment.''' (and your retirement).

<div class="figure">

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:image-20241117013839490.png
</gallery>

</div>
<div class="figure">

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:image-20241117014642043.png
</gallery>

</div>
<span id="suggested-electronics-5-10-year-old-receiver-with-opticalcoax-in."></span>
=== Suggested electronics: 5-10 year old receiver with optical/coax in. ===

There are two ways in the affordable, consumer realm to transfer digital audio signals.

* SPDIF using an optical cable(toslink).
* SPDIF using a coaxial cable.
** This is like an analog RCA audio cable, same connector, but requires a cable that is manufactured to much stricter specifications.

Getting a receiver that supports ''both'' gives you flexibility in case your motherboard only supports one or the other.

Laptops rarely have coaxial or optical out. While you can get an audio interface, this is extra money, and often not immediately available. A USB to spdif device requires an online order, while a 1/8” to RCA cable is available everywhere.

This device above allows you to have the flexibility to use whatever works best for you at the time. [https://www.ebay.com/itm/156102995033?_trksid=p4375194.c101800.m5481 Best of all, it’s $187]. Used devices like these in good condition from harman kardon(before they cheaped out, go back to 2012-ish era), denon, onkyo, etc. are being sold sub-$200 on eBay &amp; audiogon every day.

<span id="understanding-room-acoustics"></span>
== Understanding Room Acoustics ==

<span id="what-makes-rooms-sound-bad"></span>
==== What makes rooms sound bad? ====

Before we get into the technical setup, let’s talk about room acoustics because it ''really'' makes a difference. Two rooms can be identical in shape &amp; size and sound completely different if one is treated and the other is not.

Some bare-walled, people call ''“echoey”''. Those aren’t '''echoes''', they’re '''early reflections'''. An '''echo''' is when you yell and then you hear it repeat back '''a second or two later'''. An '''early reflection''' is when you speak and you hear yourself alongside yourself '''a few milliseconds later'''.

What that means is that you’re not just hearing you, you’re hearing you alongside something else. It creates a totally different sonic experience &amp; it’s annoying. It’s distortion; it’s noise added to the original signal.

<span id="why-the-word-audiophile-is-a-joke"></span>
==== Why the word “audiophile” is a joke ====

Self proclaimed '''“audiophiles”''' will spend $1000 on cables and $5000 on digital to analog converters that claim they reduce inaudible distortion 0.001%. Not '''ACTUALLY''' reduce distortion 0.001% - '''''CLAIM TO''''' reduce distortion 0.001%.

Yet, they won’t spend a few hundred dollars on room treatment that reduces distortion 5% to 15%.

It’s ridiculous. Walk into many hi-fi dealers and they won’t even mention room treatment, or try to sell you room treatment. But they will upsell you to a $4000 amplifier, or $500 cable, when it sounds the same as a $200 amp and a $5 cable.

<span id="buying-acoustic-panels."></span>
=== Buying acoustic panels. ===

<span id="acoustic-panels"></span>
==== Acoustic panels ====

24”’x48”x2” acoustic panels are the most common. Something like [https://www.atsacoustics.com/item--ATS-Acoustic-Panel-24-x-48-x-2--1001.html this ATS Acoustic 24 x 48” x 2” panel]. Pretend you’re playing pool and put the panels where the sound is going to bounce around your room as it leaves your speakers. Hang these about 2 to 3 feet above the floor, behind the speakers, and behind the listening position as well.

<span id="bass-traps"></span>
==== Bass traps ====

Bass traps are just bigger acoustic panels. The more insulating material, the lower the frequencies they absorb. It is most obvious when midrange and high frequencies are reduced in volume as this is within the range we are most sensitive to hearing differences in, since these are the frequencies of the human voice.

You may wonder what the point of a bass trap is. Most people want '''''more''''' bass, not less! Reflections are the enemy of bass.

If low frequency reflections are only a few milliseconds away from the original sound, they can cause phase issues where they cancel each other out, resulting in giant peaks and nulls in certain areas of the room, at certain frequencies. By '''absorbing''' the reflections, you wind up with '''more, and higher quality ''net''''' bass.

Bass traps usually start at 4” of thickness.

<span id="acoustic-foam"></span>
==== Acoustic foam ====

[https://www.amazon.com/JBER-Acoustic-Soundproofing-Resistant-Treatment/dp/B08R1JFZCF?crid=2ZD3L0YJRKWHH&dib=eyJ2IjoiMSJ9.I2VrY2tWcVyr8K7lX_3vj_F0oqWJSMyQWdmwc_CLXm0ETvcT70DlUkA5WkaV2_4e_k_vYbWY2ZzKwXM94vQYxgrmZOEtmIDK-ATi3sAbLLYU13HM_8tlJJ5YLp40IQciG0-1A2epdNltNct6tKiHIJ0rZZk5wdp96msp-Hhbxa7VsW81O_d2UZYfJVtqA8Hygwnzl2o2Gv5HuBsWLYUbJk3kbXaxxJEVoehDBHwgiUmRJnsIyBjr7JskLQHM8ra6AX4UsgFd3fpEZDRHMgBkpsJFDGFb8DdVepF0ODt6Q6I.4-RG_jpCaEIScDOr8y8kaImu864LS0mPU33Upfd76Zk&dib_tag=se&keywords=acoustic%2Bfoam&qid=1731827043&sprefix=acoustic%2Bfoam%2Caps%2C130&sr=8-6&th=1 Acoustic foam] is a much cheaper alternative unless you’re getting ripped off buying Auralex. However, it is way less effective. It mostly absorbs high frequencies, and the darker colors are exceptional at making rooms look depressing.

Still, this is considerably better than having nothing at all.

they don’t absorb low frequencies, just the higher ones. Compare that to, which are much more effective.

* '''Cheap Foam''': Absorbs only high frequencies; ineffective for bass.
* '''Owens Corning 703''': Absorbs a broader range of frequencies, including low ones.

Even with subpar treatment, you avoid some early reflections that can muddy up your sound. But trust me, investing in quality acoustic panels is worth it.

<span id="make-your-own-acoustic-panels"></span>
=== Make your own acoustic panels ===

Here’s what you need to make your own acoustic panels:

* '''[https://www.acoustimac.com/acoustic-insulation-materials/acoustic-insulation/owens-corning-acoustic-insulation/oc7032 Owens Corning 703 fiberglass]''' for the absorption material
* '''2x4 wood''' to frame the fiberglass.
* '''Burlap''' to hold the fiberglass in place and keep it from falling out
* '''Staplegun''' to attach the burlap to the wood frame.
* '''Brackets and drywall/brick anchors''' to hang them on your wall

That’s it. My original acoustic panels were all DIY.

Materials like Owens Corning 703 fiberglass and burlap to create broadband acoustic absorbers. Avoid using generic insulation from Home Depot as it’s not designed for sound absorption. Insulation is too loose, the sound waves move around the fibers but don’t get absorbed into it.

To recap:

* Buy some Owens Corning 703 fiberglass or a similar product for sound absorption.
* Grab some 2x4s and cut them to 2 feet by 4 feet.
* Purchase burlap and use a staple gun to wrap the fiberglass in burlap.

<span id="hunting-for-deals"></span>
== Hunting for Deals ==

To stretch your budget:

* Browsing eBay and audiogon regularly for deals on high-end speakers and receivers
* Create bookmarks for searches that fit your criteria for models and brands you like on these websites. Check it every morning for a few seconds so you get a good deal before someone else buys it.

With patience, you can put together a hi-fi system that outperforms setups priced at $10,000-$15,000 for just about $1,500. Or a setup for $400 that sounds closer to $4000.

<span id="sourcing-your-content-4k-blurays-with-the-right-drive"></span>
== Sourcing your content; 4k blurays with the right drive ==

Because the MPAA &amp; RIAA are a bag of dicks, they have managed to get almost every bluray drive manufacturer to not allow you to make a backup of your own property. They won’t rip 4k blurays.

However, there is a way around this; get a Pioneer BDR-2213 running a [https://www.avsforum.com/threads/ripping-uhd-4k-discs-with-makemkv-instructions-how-to.2942740/page-233 nice old firmware].

With this, you can rip your content in the '''exact same uncompressed quality''' you got it in. This will look so much better than the garbage low bitrate streaming quality you get from modern streaming services.

Modern streaming services give you three options:

# Use an HDCP compliant processor, HDCP compliant monitor, HDCP compliant operating system, to watch content you paid for. Jump through more hoops when PAYING than you do when pirating.
# Use a smarttv, a device that is [https://www.consumerreports.org/electronics/privacy/how-to-turn-off-smart-tv-snooping-features-a4840102036/ honest to your face that it spies on you and sells your personal data]
# Be stuck viewing a 1-2 mbps low resolution stream.
# Give a giant middle finger to the MPAA with a Pioneer BDR-2213, running old firmware.

Option 4 wins every time.

<span id="elegant-home-theater-pc-setup-for-people-who-dont-want-a-disorganized-mess"></span>

= Elegant Home Theater PC Setup: for people who don’t want a disorganized mess =

<span id="why-my-setup-makes-no-sense"></span>
== Why my setup makes no sense ==

My setup is very strange. It’s disorganized, unwieldly, and not visually appearing.

'''My setup:'''

<div class="figure">

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:image-20241201165246911.png
</gallery>

</div>
You likely want something that looks more like this:

'''Sensible setup:'''

<div class="figure">

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:image-20241201164907705.png
</gallery>

</div>
<span id="my-weird-computer"></span>
=== My weird computer ===

I don’t have a bedroom computer, home theater computer, office computer, etc. I have one computer that sits in my living room that I use for everything. I lived in an 1100 sq ft studio apartment for twelve years, so I had one PC for my 1 room home. This cube was my work computer, my video editing machine, my personal machine, my home theater PC; all in one.

'''What makes a good home theater PC is not what makes a good video editing workstation.''' For a home theater PC, you should have something like this:

* Very '''quiet'''
* Very '''cheap'''
* '''Pre-built''', because you have enough on your plate than to take time building a custom computer that rips blurays &amp; runs a pretty version of VLC
* '''Optical audio output''' If you don’t want to buy an external audio interface separately
* '''Power efficient''' so you aren’t taking 150-250 watts to play an mkv file
* '''Small pretty form factor''' that fits in with your living room perfectly

'''Above all, you don’t want it to look like a giant mess!'''

<div class="figure">

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:image-20241201163928279.png
</gallery>

</div>
<div class="figure">

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:image-20241201163719583.png
</gallery>

</div>
<span id="my-stoneage-home-theater-software"></span>
==== My stoneage home theater software ====

I showed you what I use; a computer file explorer to browse to my video &amp; music files, and double click them to play them in VLC. There are several reasons this is horrible:

* '''File &amp; folder browsing.'''
** If your folders are a mess, it will be difficult to find your stuff.
** Immich tags photos by face &amp; description; we want something like this that’ll just make sense of our 160 terabytes of stuff.
* '''Blinding user interface'''.
** Computer operating systems are designed for use with a monitor right next to you, not a TV that is 3 meters away.
** You can change your display settings &amp; scaling, but making it work with a TV makes it awkward.
* '''Manual lookup of info''': finding ratings, credits, other info isn’t immediately accessible &amp; requires leaving the file explorer or VLC to find.

There’s nothing inherently '''''wrong''''' with this setup. It’s just not everyone’s cup of tea, so we’re going to set up something built for a home theater living room system. This can be done '''quickly and easily''' - unlike many other things in GNU/Linux!

<div class="figure">

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:image-20241201163459503.png
</gallery>

</div>
<div class="figure">

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:image-20241201163452149.png
</gallery>

</div>
<div class="figure">

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:image-20241201163157029.png
</gallery>

</div>
<span id="beautiful-software-made-for-a-living-room-television"></span>
==== Beautiful software made for a living room Television ====

[https://kodi.tv/ Kodi] is a program that turns your computer into a polished home theater system for your living room TV.

* '''User-friendly interface''' designed for couch viewing. No need to squint or strain your eyes.
* '''Automatic library organization.''' Kodi scans folders &amp; files and turns the biggest messes into beautifully organized library of movies, shows, and music.

* '''Metadata integration.''' Kodi grabs information from online databases &amp; shows detailed summaries, artwork, &amp; ratings for movies, tv, and music.
* '''Open-source and offline-friendly.''' You can run Kodi without an internet connection, ensuring your ''legally ripped, totally un-copyrighted'' media collection remains private.

* '''Built-in song lyric support.''' Kodi automatically fetches &amp; displays lyrics for your music.
* '''Seamless playback with buffering.''' Kodi caches files so your media doesn’t stutter or skip, even if your server is slow or under heavy load.
* '''Effortless 4K playback.''' From high-bitrate h.264 to h.265, VC-1, or MPEG-2 files, Kodi can play anything you’ll encounter on the high seas or your personal bluray collection.

<span id="kodi-takes-minutes-to-install-configure"></span>
==== Kodi takes minutes to install &amp; configure ====

[https://kodi.tv/ Kodi] software is made for a home theater PC; you on the couch, television eight feet away, &amp; it can be installed in 2 minutes or less using a GNU/Linux distribution called [https://libreelec.tv/ LibreELEC]. This is not a convoluted installation process. It’s so seamless you’ll almost forget you’re using open source software.

<div class="figure">

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:image-20241201154938170.png
</gallery>

</div>
<span id="doesnt-my-tv-already-do-this"></span>
==== Doesn’t my TV already do this? ====

You should be able to trust your television to play television and movies. That is what it is there for.

The year is 2025, and consumer protection in the United States''(&amp; many other countries)'' is a joke. Many modern televisions come pre-configured to sell your personal data, equipped with the ability to tell who you are and what you’re watching. LG is [https://privacy.us.lg.com upfront about it]. You will hear the argument that this is necessary to keep televisions affordable; this is made by simps for television manufacturers, or the television manufacturers themselves. Above you’ll see an image of the menu of an LG G3 OLED television. '''The LG G3 OLED television is configured, by default, to spy on &amp; sell the personal information of its user; even when purchased new, at full $3600 MSRP from an Authorized LG Dealer. You cannot remove these elements of its operating system. You thought you owned the television that you bought, but the television thinks it owns you'''.

You can use your television to play back media, but it is often highly restricted. Combine this with the fact that most, if not all, modern televisions come with spyware pre-installed that you cannot remove, and we’re not doing that. My television will go on the internet over my dead body.

<div class="figure">

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:image-20241201171221209.png
</gallery>

</div>
<div class="figure">

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:image-20241201175526385.png
</gallery>

</div>
<div class="figure">

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:image-20241201181227051.png
</gallery>

</div>
<div class="figure">

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:image-20241201180622550.png
</gallery>

</div>
<span id="an-asus-asustor-flashstor-mini-pc-for-a-home-theater-computer"></span>

=== An ASUS Asustor Flashstor mini-pc for a home theater computer ===

This machine fits all of our above requirements above ''beautifully.''

<span id="quiet-operation"></span>
==== Quiet Operation ====

Dealing with noise is important when setting up a home theater PC. Your gaming PC probably sounds like an annoying $20 amazon drone, and many minipcs aren’t much better. This machine makes little to no noise even when playing back high bitrate h.265 files &amp; fits easily inside a TV stand or on a small shelf.

<span id="impressive-storage-capacity"></span>
==== Impressive Storage Capacity ====

We are using our server for storage, not the Asustor Flashstor mini-pc. If you wanted to try using this as a small starter server, here’s where the ASUS Asustor MiniPC shines—'''storage capacity'''. Unlike most if not all mini-PCs which offer 1 or 2 slots at best for SATA/NVMe drives, the asustor has '''six NVMe slots on the cheapest model'''. This lets you to install up to 24 terabytes of incredibly fast storage on the cheap asustor, or 48 terabytes on the higher end models.

<span id="cost-300-400-on-ebay"></span>
==== Cost: $300-$400 on eBay ====

These can be found under [https://www.ebay.com/itm/126778026449?_skw=asustor&itmmeta=01JE2ADEGT8FRQ5A3F382XAQFR&itmprp=enc%3AAQAJAAAA4HoV3kP08IDx%2BKZ9MfhVJKlHyaaNfhwG0a0qDGiezVnDGbOrDK%2F0m3Z9BOZntIaopaUpFxI2BZ%2FT%2FUTiynBT9r7jGvcZTsYZmKJPAsYnqOf9l1H4iDCewdnk0vpdINtJo5cipLcikC049ecEiax%2FSE5Kafw1PFAajDKRAWEloFrPrK8tfztOIe7j8c9yBUCD17X8rMaY8gUt0KDZBg%2BGT7oGU3%2FHCKycRaUA7HfxYxgfYWQ501010hLK2fFmdl4uPjbvGSslrE0lW3RuxeEVFXNlA331a1QtuPZO%2BPMydshy%7Ctkp%3ABk9SR7rotcrwZA $350 used] which gets you a lag-free, quiet machine with six NVMe slots.

<span id="audio-video-output-options"></span>
==== Audio &amp; Video output options ====

The Asustor Flashstor does '''4k60 out of its HDMI port''' just fine. Some cheap no-name fly by night minipc companies use old HDMI standards for their ports &amp; get stuck at 4k30.

For high-quality audio, having an '''optical SPDIF''' output is important. as mentioned before, this allows the digital-to-analog conversion to be handled by dedicated audio equipment rather than your multipurpose PC, which sidesteps the noisy nonsense you get when you try encoding video or doing CPU intensive things with headphones plugged in. You may not notice this while your headphones are turned up as you’re engaged playing an exciting game, but quiet passages of movies get ruined by this very easily. The Asustor flashstor includes an optical SPDIF audio output jack, allowing you to connect directly to most modern home theater receivers.

* '''Benefit''': Avoids the need for additional USB audio interfaces.
* '''Setup''': Use a simple $5 optical cable from Walmart to connect to your stereo system.

It has '''optical audio output'''. This allows you to plug the machine into a '''stereo receiver'''’s optical audio input, a discrete '''digital to analog converter''', or an '''integrated amp'''’s optical audio input for clean sound output.

HDMI carries audio, but if you’re like me &amp; have a separate audio setup from your television, you’d have to get an HDMI audio/video splitter to get HDMI video to your TV and SPDIF digital audio to your stereo receiver that goes to your speakers. Some TVs can pass through the audio digitally to your stereo receiver, some don’t, but even if they do this is an added pain in the ass. Having optical audio out makes this easier.

<span id="powerful-expandable-machine"></span>
==== Powerful, expandable machine ====

Even the cheapest Asustor flashstor handles 4K video effortlessly. [https://www.aaawave.com/asustor-flashstor-6-gen-2-6-bay-nas-quad-core-2-3ghz-cpu-10gbe-ports-8gb-ram-ddr5-6x-m-2-ssd-slots-diskless-fs6806x/ Higher end models] are '''twice as powerful as the server in this guide''' and only take a '''fraction of its power''', making them suitable as a starter server. Low end models have '''6 NVMe solid state drive slots''', but you can buy this with up to [https://www.asustor.com/en-gb/product?p_id=91 12 NVMe drive slots] which would give you 48 terabytes of NVMe storage for a server, with 10 gigabit ethernet for fast network transfers.

<span id="the-asustor-flashstor-can-be-a-starter-server."></span>
=== The asustor flashstor can be a starter server. ===

Using an Asustor as a starter server is a great idea. If you know you want a home theater PC, you’re going to buy something like this anyway; and even the low end model is powerful enough for most tasks. You can always demote it to a home theater PC down the line when/if you decide to put together a giant 200 terabyte monster like what is pictured above. like what I have pictured above.

<span id="dont-use-your-server-as-an-htpc-at-the-same-time-attack-surface-why-you-should-care"></span>
==== Don’t use your server as an HTPC at the same time; attack surface &amp; why you should care ====

The '''attack surface''' (or threat surface) refers to all the different points where a hacker could potentially gain unauthorized access to your system. This means that the more you install onto your machine, the greater the likelihood you turn into one of the poor schmucks in <code>/r/asustor</code> who got [https://www.reddit.com/r/asustor/top/?t=all owned by ransomware]. The more things a machine does, the larger its attack surface becomes &amp; the more opportunities attackers have to exploit vulnerabilities.

If you use the same PC for Kodi ''and'' services like Mailcow (mail server), FreePBX (phone system), Immich (photos), or Nextcloud (notes), you’re mixing a '''home theater interface''' with '''mission-critical infrastructure.''' Bad idea.

<span id="why"></span>
===== Why? =====

# '''Increased Exposure:''' Running Kodi means more risk of vulnerabilities from media files, plugins, user interaction, etc. If exploited, it could compromise your entire server &amp; everything running on it.

# '''Conflicting Security Needs:''' A server for mail and photos requires high uptime, strict access control &amp; limited exposure. A home theater PC is inherently less secure because it’s meant to interact with more devices, networks, &amp; potentially risky media.
# '''Damage Scope:''' If someone hacks your Kodi system, do you really want that person having backdoor access to your email, phone, or photos? Keep the two separate &amp; isolate them for better security.

<span id="why-not-use-it-as-a-router"></span>
==== Why Not Use It as a Router? ====

You might wonder, can your MiniPC double as a router since it has two Ethernet ports? These are 2.5 GbE ports, which is faster than the typical 1 GbE ports. It offers speeds of 250 to 290 MB/s. However, they use Realtek chipsets (likely the RTL8169) &amp; while you '''can''' use Realtek for a firewall, you really '''shouldn’t'''. This isn’t a meme like running your own self managed mail server. It’s just a bad idea. Don’t ever mix Realtek chipsets with FreeBSD based firewalls(which pfSense is).

<blockquote>'''IMPORTANT NOTE:''' Avoid using Realtek chipsets for firewall purposes. Stick to using your MiniPC as a home theater PC instead.
</blockquote>
<span id="being-silly-adding-eight-3.5-enterprise-class-hard-drives-to-the-asustor-flashstor-mini-pc."></span>
==== Being silly: adding eight 3.5” enterprise class hard drives to the Asustor Flashstor mini-pc. ====

Let’s say you chose to use this device as a server down the line. It only has NVMe slots for solid state drives. 24 terabytes of flash storage might be too little for you. If you want to use hard drives with it, you can’t plug desktop drives directly into it; but that doesn’t mean you can’t try. :)

You can actually add eight 3.5” desktop hard drives to an asustor flashstor if you bought one with a '''USB-C 4.0 port'''. If you’re looking to expand beyond NVMe, the higher end models with USB-C ports allow this. If you wanted to go crazy, you could get the following hardware. To be clear, this is ridiculous &amp; not recommended; but there’s something fun about doing ridiculous things. The lengths I have gone through to make use of hardware I already own are great, and I feel compelled to share some of what is possible with you.

* '''USB-C to PCI Express Card enclosure''': [https://www.startech.com/en-us/usb-hubs/2tbt3-pcie-enclosure This unit] allows you to plug a desktop PCI Express card slot into a computer that has a USB-C port. This is needed since the flashstor has no PCI Express card slots fit for desktop PCI Express cards. You might have to cut a hole in it for the SATA cables to come out of.
* '''PCI Express Serial ATA card:''' [https://www.ebay.com/itm/235464441248?_skw=Intel+RS3WC080&itmmeta=01JE2DYJ39X6E1NKT2GP1AS8V9&hash=item36d2c63da0:g:CCUAAOSwV-pmX3xi&itmprp=enc%3AAQAJAAAA8HoV3kP08IDx%2BKZ9MfhVJKkYGG99u%2FPOyPVfli1VueephfY%2BGqf7itGPMgXK9xShe8TO%2F13dLnEVBooi09dW9ucYYBmmKuUWb%2Bklu5F7ZOnOqT4ElYFOD5WYW%2BEopRsmt5d%2FkPeRRw017E%2BNG9Vw314EAe2bRQy6uCoaUvPIN8kyPH9KL4MntdQwmCFrwfF5uxIhLfnNUaA9I7KetS1rB%2BFQD9R2XPt0jqfIoa6Zm5MMxsDO1uvhf7Pj1CpfOr4sI6KcjRjJGboW3btGqsWtVMbkzNJJ6gIge4pvstvwwIwf3U8WSiTkw1aDplebZRVtYQ%3D%3D%7Ctkp%3ABk9SR9yh-s3wZA This] lets you plug in another 8 serial ATA desktop hard drives.
* '''Mini SAS to SATA cable, SFF-8643:''' An [https://store.supermicro.com/us_en/supermicro-minisas-hd-to-4-sata-30-60-70-70cm-cable-cbl-sast-0704.html SFF-8643 adapter cable] goes between your PCI Express SATA card and your eight hard drives. You would need two.
* '''Power Splitter''': Needed for powering multiple drives, something like the [https://www.startech.com/en-us/cables/pyo4sata startech SATA power splitter].
* '''SATA drive power supply:''' You’d now need to power those SATA drives.
** Something like [https://www.amazon.com/Warmstor-Adapter-Computer-Connector-Converter/dp/B076WZ1N4K/ref=sr_1_1?crid=37QM5VOFZAM1Q&dib=eyJ2IjoiMSJ9.at93EUUqXo4LGsFjaIg29R-3dDWigqy5MVKmMfIc25yVZWtMqLDzaSH-DHMJYveHuMrs1bMeVSNTqIfDEcuR4zNnfmb44SYNvSe9ZIwYayzkeF5ujjqWWJoOQUxT8etauoS_V66crJBIkf0DITtJN6fJdcnduzoipt7rnRagWy14VBViEpoWeBAO0Fyfht6AfmB3Xn6ymUIyFiie2_j5yQOP5wGBtK2ng7o6GjnRfvY.7fqScExFcxLobhy3oP0mk34glrLTeHSzt46MPMK8Ro4&dib_tag=se&keywords=Sata+Power+Supply&qid=1733101321&sprefix=sata+power+supply%2Caps%2C281&sr=8-1 this] could power 2 drives at a time.
** Any PC power supply that can do over 10 amps on the 12 volt rail would suffice for eight 3.5” enterprise class serial ATA hard drives, but you see why this is getting silly.
** Either you are going to have to do some research to find a sleek looking power supply that does 10 amps at 12 volts to reliably power eight 3.5” enterprise class hard drives, OR:
** Short the green PS_ON wire on a desktop PC power supply to the black wire with a paperclip to turn it on. Desktop PC power supplies only turn on when they are plugged into a desktop computer, and this would only be plugged into the drives.

<span id="setting-up-your-home-theater"></span>
== Setting Up Your Home Theater ==

<span id="introduction-once-you-have-your-mini-pc"></span>
=== Introduction once you have your mini-pc ===

'''Overview of steps involved:'''

[https://libreelec.tv/ LibreELEC] is a GNU/Linux distribution that takes '''less than 90 seconds to install''' that starts up into [https://kodi.tv/ Kodi]''(our media center software)'' software '''right out of the box.''' This is so easy; it just works. It’s so good you’ll forget you’re even using GNU/Linux or open source software. The steps below are as follows:

# '''Make LibreELEC install disk''' to install LibreELEC linux distribution onto our asustor home theater PC.
# '''OPTIONAL: Install NVMe drives into Asustor.''' My home theater PC does not store any content; that is what my server is for. If you want your Asustor to have '''local storage''' as well, you can install NVMe drives into the bottom of it.
#* Use a phillips #0 screwdriver to remove the four screws on the bottom of the Asustor.
#* Be gentle; the clips you have to pull back to fit your NVMe drive in aren’t the most durable. In fact, they remind me of the flimsy MacBook A1181 screen bezel clips that broke if you looked at them the wrong way.
#* Avoid pressing directly on the SSD’s chip when pushing it into the mini-PC. Instead, apply pressure to the pc board of the solid state drive so you don’t put pressure on the solder balls under the SSD’s chip.
#* If at any point you are debating whether to pull back harder on the clips of the asustor that hold the NVMe drive in, or to push harder on the SSD, '''always elect to pull harder on the NVMe clips on the asustor'''. The cost of those breaking is nothing; just use a piece of [https://www.amazon.com/VCHOMY-Sublimation-Polyimide-Temperature-Insulation/dp/B0C9PMHWGS kapton heat resistant tape] to hold the SSD in. The cost of breaking the SSD, is several hundred dollars, or random reboots if you cause a crash in a solder ball that will take you months to trace back to that stupid SSD.
# '''Plug Asustor into television, keyboard, &amp; mouse'''
# '''Disable secure boot/security features in asustor BIOS(UEFI technically)''' so we can install Linux on it.
# '''Erase Bloatware:''' Asus’s garbage software will be removed so it can never be used again, even by accident.
# '''Install Libreelec:''' This provides a clean, efficient operating system specially made for home theater PCs.
# '''Set Up KODI:''' We’ll use this to catalog media files making them easy to search &amp; access. As soon as you turn the computer on, in less than 30 seconds it will be booted up into KODI so you can access all of your files.

<blockquote>'''Note:''' This setup will automatically pull information from internet databases, giving you detailed descriptions &amp; reviews of your content.
</blockquote>
<span id="installing-libreelec-operating-system-with-kodi"></span>
== Installing LibreELEC operating system with KODI ==

# '''Download LibreELEC:'''
#* Head to the [https://libreelec.tv/downloads/generic/ LibreELEC] website and download the generic image for your hardware.
#* Generic is what we want; you can download versions for other non-x86 architectures if you want a home theater PC that isn’t based on x86, which is cool, but we’re using an x86 based minipc here.

<span id="step-1-creating-a-bootable-libreelec-usb-drive"></span>
=== Step 1: Creating a Bootable LibreELEC USB Drive ===

<span id="download-libreelec"></span>
==== 1.1 Download LibreELEC ====

# Head to the [https://libreelec.tv/downloads/generic/ LibreELEC Downloads] page and download the generic image for your hardware.
#* Choose the '''Generic''' version for x86-based systems.
#* If you’re feeling adventurous, you can download versions for non-x86 architectures, but we’re focusing on an x86-based mini-PC here.
# The file will be in <code>.img.gz</code> format. You will need to unzip it.


-----

<span id="step-2-unzip-the-.gz-file"></span>
=== Step 2: Unzip the <code>.gz</code> File ===

'''Instructions for GNU/Linux, macOS, and Windows:'''

<ul>
<li><p>'''Linux:'''</p>
<pre>gunzip LibreELEC-Generic.x86_64-12.0.1.img.gz</pre>
<p>This will extract <code>LibreELEC-Generic.x86_64-12.0.1.img</code> in the same directory.</p></li>
<li><p>'''macOS:'''</p>
<ol style="list-style-type: decimal;">
<li><p>Open Terminal and navigate to the directory with the downloaded file:</p>
<pre>cd /wherever/you/downloaded/it/to</pre></li>
<li><p>Use the <code>gunzip</code> command:</p>
<pre>gunzip LibreELEC-Generic.x86_64-12.0.1.img.gz</pre></li></ol>
</li>
<li><p>'''Windows:'''</p>
<ol style="list-style-type: decimal;">
<li>Download and install a tool like [https://www.7-zip.org/ 7-Zip].</li>
<li>Right-click the <code>.gz</code> file and select '''7-Zip → Extract Here''' to extract the <code>.img</code> file.</li></ol>
</li></ul>


-----

<span id="step-3-create-a-bootable-usb-drive"></span>
=== Step 3: Create a Bootable USB Drive ===

'''⚠ Warning:''' This process will erase everything on the USB drive.

# Insert a USB flash drive (at least 4GB in size) into your computer.
# Use one of the methods below to write the LibreELEC image to the USB drive.

<span id="windows-1"></span>
=== Windows: ===

# Download and install [https://rufus.ie/ Rufus].
# Open Rufus and select your USB drive.
# Click the '''“SELECT”''' button and choose the <code>.img</code> file you extracted.
# Click '''“Start”''' and let Rufus create the bootable USB.

<span id="macoslinux"></span>
=== macOS or GNU/Linux: ===

<span id="figure-out-which-is-the-right-usb-drive"></span>
==== Figure out which is the right USB Drive: ====

<ol style="list-style-type: decimal;">
<li><p>Open the terminal and run:</p>
<pre>sudo fdisk -l</pre></li>
<li><p>Make a note of the connected drives.</p></li>
<li><p>Insert your USB flash drive and run the command again:</p>
<pre>sudo fdisk -l</pre></li>
<li><p>Identify the new drive that appears. It’s usually something like <code>/dev/sdX</code> or <code>/dev/diskX</code>.</p></li>
<li><p>'''Double-check''' that you’ve identified the correct drive:</p>
<ul>
<li>Unplug the USB drive.</li>
<li>Run <code>sudo fdisk -l</code> again. The drive should disappear.</li>
<li>Plug it back in and confirm it reappears.</li></ul>
</li>
<li><p>If you’re sure the drive is correct, proceed.</p></li></ol>

<span id="write-the-image-to-the-usb-drive"></span>
==== Write the Image to the USB Drive: ====

<ol style="list-style-type: decimal;">
<li><p>Replace <code>/dev/sdX</code> with your USB drive’s path and run:</p>
<pre>sudo dd if=LibreELEC-Generic.x86_64-12.0.1.img of=/dev/sdX bs=4M status=progress</pre></li>
<li><p>Wait for the process to complete. It may take a few minutes.</p></li></ol>


-----

<div class="figure">

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:image-20241201233219809.png
</gallery>

</div>
<div class="figure">

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:image-20241201233359161.png
</gallery>

</div>
<span id="step-3-set-up-the-asustor-minipc"></span>
=== Step 3: Set up the Asustor minipc ===

'''Connect to Your TV and Network:'''

* '''HDMI Cable:''' Connect it from the mini PC to your television.
* '''Ethernet Cable:''' Connect an ethernet cable so it can connect to your server’s ZFS pool.
* '''Optical Audio Cable:''' Use this for audio output to your stereo system. Make sure you insert the optical cable correctly; it is not like a USB-C cable, it fits one way, and there are four possible ways for this to go in. That gives you a 25% chance to plug it in without destroying the jack if you are blindly messing around with it trying to plug it in. Those are bad odds. Pay attention to the plug &amp; the jack!
* '''Power Cable:''' Plug this in last, as the asustor flashstor minipc powers on automatically when connected.

<div class="figure">

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:image-20241201194913746.png
</gallery>

</div>
<div class="figure">

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:image-20241201215358549.png
</gallery>

</div>
<span id="step-4-boot-into-libreelec-installation-and-install-it"></span>
=== Step 4: Boot into LibreELEC installation and install it ===

# Insert the bootable USB drive into your mini-PC.
# Restart the system and enter the BIOS/UEFI settings by pressing '''F2''' over &amp; over again as fast as possible right after the machine turns on.
# Go to '''“boot”''' menu by using the right arrow key and pressing enter.
# Set the USB drive as the primary boot device.
# In the BIOS, disable any TPM and secure boot options that interferes with Linux installation. This is similar to what we did on the Intel NUC early in the guide when installing pfSense onto it.
# Save changes &amp; reboot. LibreELEC will boot from the USB drive. Hitting '''F10''' will exit the BIOS &amp; save your changes.


-----

If you managed to erase your entire computer by writing the LibreELEC image to your operating system drive EVEN AFTER reading these instructions, congratulations! You’re almost as stupid as me. Almost. Don’t do that.

<div class="figure">

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:image-20241201220116461.png
</gallery>

</div>
<div class="figure">

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:image-20241201225302004.png
</gallery>

</div>
<div class="figure">

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:image-20241201225320577.png
</gallery>

</div>
<div class="figure">

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:image-20241201225331606.png
</gallery>

</div>
<span id="step-5-install-libreelec-onto-the-asustor"></span>
=== Step 5: Install LibreELEC onto the Asustor ===

We are erasing all of the Asustor software &amp; replacing it. This process will take less than 90 seconds. and

Next, we install LibreELEC, which is just enough OS to run <code>Kodi</code>.

# '''Boot and Install''': Follow the prompts to install LibreELEC onto the internal eMMC.
# Choose the drive you wish to install it onto, which will be the <code>/dev/mmcblk0</code> device in the case of the Asustor Flashstor. That is the memory that the ASUS software is installed onto; we are erasing it to install LibreELEC &amp; KODI.
# You’re done. That’s it. In &amp; out in less than 90 seconds - amazing. :)

<blockquote>'''NOTE:''' If you have not installed any new NVMe drives into the Asustor Flashstor minipc, there should only be one device showing up to install onto, which will be the internal EMMC at <code>/dev/mmcblk0</code>. If you have installed new NVMe SSDs, they will show up qith <code>/dev/nvmexn1</code> notation with <code>x</code> being the number of the SSD in the machine.
</blockquote>
<div class="figure">

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:image-20241201230528292.png
</gallery>

</div>
<div class="figure">

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:image-20241201230547810.png
</gallery>

</div>
<div class="figure">

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:image-20241201230624240.png
</gallery>

</div>
<div class="figure">

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:image-20241201230635545.png
</gallery>

</div>
<div class="figure">

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:image-20241201230715169.png
</gallery>

</div>
<div class="figure">

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:image-20241201230801939.png
</gallery>

</div>
<div class="figure">

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:image-20241201231031498.png
</gallery>

</div>
<div class="figure">

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:image-20241201231050869.png
</gallery>

</div>
<div class="figure">

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:image-20241201231926114.png
</gallery>

</div>
<div class="figure">

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:image-20241201231959756.png
</gallery>

</div>
<div class="figure">

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:image-20241201232226059.png
</gallery>

</div>
<div class="figure">

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:image-20241201232445242.png
</gallery>

</div>
<div class="figure">

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:image-20241201232501935.png
</gallery>

</div>
<div class="figure">

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:image-20241201232537212.png
</gallery>

</div>
<div class="figure">

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:image-20241201232606886.png
</gallery>

</div>
<div class="figure">

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:image-20241201232625879.png
</gallery>

</div>
<div class="figure">

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:image-20241201233609572.png
</gallery>

</div>
<div class="figure">

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:image-20241201233509199.png
</gallery>

</div>
<div class="figure">

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:image-20241201233621905.png
</gallery>

</div>
<div class="figure">

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:image-20241201233647328.png
</gallery>

</div>
<div class="figure">

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:image-20241201233736391.png
</gallery>

</div>
<div class="figure">

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:image-20241201233907464.png
</gallery>

</div>
<div class="figure">

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:image-20241201233943591.png
</gallery>

</div>
<div class="figure">

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:image-20241201233658326.png
</gallery>

</div>
<div class="figure">

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:image-20241201234055519.png
</gallery>

</div>
<span id="step-6-boot-into-the-libreelec-system-set-it-up"></span>
=== Step 6: Boot into the LibreELEC system &amp; set it up ===

After installing LibreELEC, it will boot into the operating system &amp; start KODI. The rest of the setup is a breeze.

'''Networking Configuration''':

* Use the default internet connection settings.
* There is no need to configure a static IP address for a ''client.'' Static IP addresses are for servers.
* If you are using this to watch stuff stored on your server’s ZFS pool, disabling samba server &amp; disabling ssh is the smart way to go. No need to run unnecessary services if you don’t have to.

'''Audio Configuration:'''

By default, it will output audio via the HDMI cable.

* If your HDMI cable connects to an '''audio/video receiver that is hooked up to your speakers''', you’re fine.
* If your HDMI cable connects to your '''television''', you may hear the audio through your '''TV speakers''', which is horrible; we will need to change where Kodi outputs to.

To change the audio output:

# '''Access System Settings''': Navigate to the gear icon for settings, then '''Audio'''.
# '''Select the Audio Output Device''': Choose <code>ALSA: HDA Intel PCH, ALC888-VD Digital S/PDIF</code>. Yours may look mildly different - we want whatever looks closest to S/PDIF digital optical/toslink output. Experiment to find which one works for you.
# '''Check Display Settings''': Make sure it is set to what your television is capable of. in my case, it is set at ***3840 by 2160 and 60 fps.

<blockquote>'''Why not a static IP? Didn’t we make a static IP for everything else?'''

Static IPs aren’t important for a computer that doesn’t provide services. When we’re running a server, like our machine with the ZFS pool that stores our media files, we are running something where clients(aka our home theater PC) are going to want to know where to access it.

Think of your server like your favorite store. We are going to tell our home theater PC to always go to the store to get movies(….) at <code>192.168.5.2</code> - so our server always NEEDS to be at <code>192.168.5.2</code>.

The home theater PC we are setting up right now is the ''“customer”'' - it doesn’t have to have a static IP, nor does it always have to be at the same address every day.

A customer can visit a store from a different address every day; it makes no difference to the shopowner selling goods to the customer. However, if the '''''store’s''''' address changed ''every single day'' without notice, the customer would have a very hard time finding the store. They may stop going to that store altogether.

We can use the default setup where the server grabs an IP address via DHCP''(aka, it grabs whatever’s available from the router)'' without concern here.
</blockquote>
<span id="step-7-adding-media-content-to-kodi"></span>
== Step 7: Adding Media Content to Kodi ==

After setup, let’s add some media content to your system.

# Click on '''“Movies”''' or '''“TV shows”''' on the side.
# Click '''“Add Videos”'''
# Click where it says '''<None>''' in order to add an address.
# '''Add Samba Share''': Use the IP address and share path to add your media content. For our server that we set up, you would use as follows to access the ZFS pool:

<ul>
<li><pre>  smb://192.168.5.2/archive</pre></li></ul>

<blockquote>'''SECURITY NOTE:''' In my personal setup, I like to make a '''separate read only''' user when setting up samba for my media directory that I use for clients that will be viewing music, videos, tv, etc.

The reason for this is that if the software I am using to view has a delete button I accidentally press, my cat walks on my keyboard/remote while I am watching something, the software has a bug/glitch etc., I do not lose my media collection. Here is an example from my own samba configuration:

<pre>[television]
    comment = television shows
    path = /drive1thru8/television
    browseable = yes
    read only = no
    valid users = louis, kodi
    write list = louis
    create mask = 0644
    directory mask = 0755
    force user = louis
    force group = louis
    inherit permissions = yes
    inherit acls = yes
    ea support = yes</pre>
This would be accessible at <code>smb://192.168.5.2/television</code>. My user, <code>louis</code>, can read &amp; write, whereas the user <code>kodi</code> can only read. I would log into the samba share as <code>kodi</code> from my home theater PC, or any client where I solely intend to view content.

Even if Kodi’s source code were hijacked by some bastard whose goal it was to destroy our entire media library, they would not be able to.

'''''LESSON:''''' '''It is good practice to give minimum necessary permissions to everything!'''</blockquote>

# '''Scan Media''': Scan the added directories for movies, TV shows, etc., and organize them in Kodi.
# '''Choose media type:''' For '''“This directory contains”''', choose the media type so Kodi is able to look things up for you about what you are watching, grab art, reviews, ratings, etc.
# Click onto Movies/TV(whatever you just added) &amp; search for something.
# Play &amp; enjoy :)

<span id="performance-testing-with-high-quality-media"></span>

== Performance Testing with High-Quality Media ==

Once the physical setup is complete, it’s time to test how well this setup handles high-definition content.

* '''Video Playback Test:''' Let’s see how it handles a 4K video file. I’m using a 70-80 GB file of ''“Batman Begins”'' to push the limits. See if it is able to seek within the file quickly, and if there’s any lagging on action scenes or very dark-shot areas ''(this is where bitrate is usually going to be highest, and therefore most difficult for cheap hardware to play back properly)''
* '''Audio Performance:''' Listen for any distortions or skipping in the digital audio output, digital scratching noises.

<span id="noise-levels-and-setup"></span>
== Noise Levels and Setup ==

My custom water-cooled desktop with Noctua fans is noisy. I provided comparisons witha DPA 4065 omnidirectional mic in the video, in a normal living room, for you to hear; between that and the Asustor Flashstor minipc. This is not a ''completely'' passive device, but for most; it does amazingly well.

<span id="piracy"></span>
= “Piracy” =

<span id="we-cant-talk-about-home-theater-pcs-without-delving-into-piracy."></span>
== We can’t talk about home theater PCs without delving into Piracy. ==

We live in a world where companies are trying to normalize the idea that you don’t own what you bought &amp; paid for anymore. Piracy is no longer an immoral act; in many cases, it is a necessity to retain what you have rightfully purchased from companies that think that word means something different than what was written in the Oxford English Dictionary 700 years ago.

<div class="figure">

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:image-20241202045458828.png
</gallery>

</div>
<div class="figure">

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:image-20241202045338954.png
</gallery>

</div>
<span id="the-death-of-digital-ownership"></span>
== The Death of Digital Ownership ==

<span id="sony-discoverys-content-removal-scam"></span>
=== Sony &amp; Discovery’s Content Removal Scam ===

Sony &amp; discovery tried to [https://web.archive.org/web/20240000000000*/https://www.playstation.com/en-us/legal/psvideocontent/ remove customer content from their libraries AFTER they purchased it]. The word PURCHASE - not rent, was used to describe the transaction.

<blockquote>Discovery Entitlements Affected Titles

As of 31 December 2023, due to our content licensing arrangements with content providers, you will no longer be able to watch any of your previously purchased Discovery content and the content will be removed from your video library.

We sincerely thank you for your continued support.

Thank you,

PlayStation Store
</blockquote>
They might as well be telling you to go gargle their balls.

<div class="figure">

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:image-20241202061005960.png
</gallery>

</div>
<span id="the-hidden-redefinition-of-purchase"></span>
=== The Hidden Redefinition of “Purchase” ===

Sony’s claim is that their terms of service were changed to '''''redefine the word “purchase” to mean something new''''' so they could still CLAIM you were “buying” something when you were not. See their [https://web.archive.org/web/20240403115117/https://www.playstation.com/en-us/legal/psn-terms-of-service/ terms of service] below:

<blockquote>'''10.1.''' All intellectual property rights subsisting in PSN Content, including all software, data, services, and other content subsisting in or used in connection with PSN, the Online ID and access to content and hardware used in connection with PSN belong to SIE, its affiliates, and its licensors. Use of the terms “own,” “ownership”, “purchase,” “sale,” “sold,” “sell,” “rent” or “buy” in this Agreement or in connection with PSN Content does not mean or imply any transfer of ownership of any content, data or software or any intellectual property rights from SIE, its affiliates or its licensors to any user or third party.
</blockquote>
<blockquote>'''10.2.''' Except as stated in this Agreement, all Content provided through PSN is licensed on a non-exclusive and revocable basis to you for your personal, private, non-transferable, non-commercial, limited use on a limited number of PlayStation Devices or other devices in the country in which your Account is registered.
</blockquote>
They use the word '''“PURCHASE”''' on their website, but then hide behind this garbage buried into page 21 of their terms of service. The word '''''“PURCHASE”''''' has had a specific meaning since the 14th century, when Oxford English Dictionary defined the word ''“purchase”'' as meaning ''“to acquire in exchange for payment; to buy”'' or ''“obtaining something in exchange for payment in money or an equivalent; buying.”''

<span id="a-history-of-anti-consumer-behavior"></span>

=== A History of Anti-Consumer Behavior ===

Because consumer protection in the United States is a joke, they are allowed to redefine the meaning of a 14th century word to justify taking away your personal property without refunding your money.

If they were honest, they would put this ''“new”'' definition of the word ''“purchase”'' on their front page next to the '''“Add to cart”''' button. '''''They don’t do that.''''' They hide it on page 21 of a legalese terms of service they know damn well you will never read.

They know what it would do to their sales if they said ''“purchase actually means we can take it back from you at any time without refunding you”'' in the same font size they use next to the '''“Add to cart”''' button. They’re not.

This is the same company that [https://en.wikipedia.org/wiki/Sony_BMG_copy_protection_rootkit_scandal installed malware &amp; rootkits on people’s computers when they legally paid for content] that expects you to be an honest upstanding citizen who buys content &amp; allows them to take it back.

Right.

<div class="figure">

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:image-20241202050157648.png
</gallery>

</div>
<div class="figure">

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:image-20241202050450042.png
</gallery>

</div>
<span id="streaming-services-paying-more-for-less"></span>
== Streaming Services: Paying More for Less ==

<div class="figure">

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:image-20241202061328214.png
</gallery>

</div>
<div class="figure">

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:image-20241202061922004.png
</gallery>

</div>
<span id="netflixs-4k-scam"></span>
=== Netflix’s 4K scam ===

Modern streaming services are equally dishonest when they try to upsell customers to a higher priced plan for higher quality video. Modern media companies are obsessed with control and want you to view and listen to content on completely locked down platforms. This is to the point where you have to build a special computer or use a television that is [https://www.consumerreports.org/electronics/privacy/how-to-turn-off-smart-tv-snooping-features-a4840102036/ blatant spyware] to watch the content you paid for in the advertised bitrate &amp; resolution.

'''I don’t use the Netflix application on my LG television to watch Netflix because my television attempts to collect &amp; sell my personal data without my consent from the moment I turn it on. I find this unacceptable.''' I am happy to pay to watch content; but I am not going to give up my data &amp; my privacy to do it, nor do I wish to trust such an unscrupulous piece of hardware that '''''opts me into this by default.'''''

Netflix will upsell you to 4k, but '''nowhere on their plans page, pricing page, or help page do they tell you that you will receive a low bitrate, 720p stream if you use firefox on GNU/Linux - or a very low bitrate 1080p stream in chrome''' The only way to get a high bitrate, 4k stream is as follows:

<blockquote>The requirements to actually get 4K streaming working on a PC are buried in documentation and frankly absurd. You need:

* Windows 10 or newer (not necessary when pirating)
* HDCP 2.2 compliant monitor and GPU (not necessary when pirating)
* GPU with HEVC hardware decoder (not necessary when pirating)
* 4K monitor (even if you just want higher bitrate, or to view 1080p content on a 1080p monitor, which is not a problem when pirating)
* 4K HDR monitor (some services won’t deliver 4K without HDR, which is not a problem when pirating)
* PlayReady 3.0 support (not necessary when pirating)
* Microsoft Edge or the Windows Store app (pirating is cross platform &amp; operating system agnostic)
* Intel processor with SGX enabled (AMD processors are unsupported, enjoy your oxidating 14th gen intel CPU; by the way, piracy is processor/platform agnostic &amp; works on all)
* No DisplayLink products or similar display adapters (piracy plays on any display product)

As someone who repairs motherboards professionally, I find it unreasonable to expect average consumers to verify all these requirements before subscribing to a service that prominently advertises “4K streaming” as a feature. For comparison, game publishers clearly list their system requirements right next to the purchase button.

Even more frustrating is that these restrictions exist purely due to DRM requirements, not technical limitations. The same content can stream perfectly fine at full quality to smart TV apps, proving the bandwidth and technology exists to deliver high-quality streams to any capable device.
</blockquote>
<span id="the-hidden-quality-gap"></span>

=== The Hidden Quality Gap ===

Just as Sony doesn’t have the balls(or the integrity) to place their re-defined concept of what it means to ''“purchase”'' something on their product page; '''Netflix doesn’t have the balls to list their series of limitations on 4k playback on the plans &amp; pricing page.'''

Netflix '''''KNOWS''''' conversions will go down if consumers understand the hoops they’d have to jump through to get a higher quality stream, ESPECIALLY if they knew that they could pay for the higher quality plan and get an even WORSE VIDEO QUALITY than they’d get on the normal plan just because they weren’t using hardware anointed by netflix to properly f the user in the ass.

In the words of Eteel from reddit:

<blockquote>Publishers advertise the requirements needed to run the game, and they do it freely—literally next to or below the button you press to buy the game. And that’s even though no one actually, realistically expects to run Cyberpunk 2077 in 4k with raytracing on GTX 1080 TI. That in no way compares to the reasonable base-level expectation that you’d be able to play a video in 4k using Chrome.

Netflix advertises no such information. They do have a help page listing which broswers support which resolution, but in order to get to it (or to even find out that it exists), you need to search for why you’re not getting 4k in the first place, and in order to search for why you’re not getting 4k in the first place, you already need to have bought the service thinking you’re going to get 4k using Chrome…

And here’s what Netflix doesn’t tell you even on this help page: they don’t tell you that while Chrome does support 1080p, it does not support high-bitrate 1080p. Playing Vikings: Valhalla on 1080p on Edge gets you 3000 bitrate while on Chrome gets you 1000 bitrate. That’s a significant difference they don’t advertise.

There’s still more to say about this, but I digress.

And to address this comment of yours:
</blockquote>
<blockquote><blockquote>What would that be? I understand criticism of DRMs but endorsing piracy as a solution for consumer issues would set a crazy precedent. Would it be okay for me to shoplift items that are too expensive for me to purchase? What if the cheaper ones quality doesn’t meet my demands?
</blockquote></blockquote>
<blockquote>It’s not okay to shoplift items that are too expensive, but it is also not okay for Ubisoft Connect to advertise buying a license to play a game as buying the game. It literally has a button that says “Buy the game,” but when you read the fine print, it tells you that you don’t actually own the game even if you buy it (in contrast to buying a shirt which you own.) In actuality, if you buy one of their games, they still withhold the right to remove it from the store, in which case you’d be unable to play it even though you bought it. In essence, they redefine the word “buy” to not mean “own” despite the fact that in common language usage, we have always understood the two terms to exist in connection to each other.
</blockquote>
Keep in mind that even the 3 mbps 1080p stream is '''garbage'''. When you pirate, you have the option to download a full bitrate video file. You can download movies &amp; television shows that have '''50 mbps bitrate with high quality encoding settings''', and often '''completely uncompressed blurays.'''

Or, you can stream a piece of media using netflix at 3 mbps. 1 mbps if you’re using the wrong web browser. Or processor. Or screen cable. Or whatever.

And download button? screw your download button - you can watch it until you can’t, and if you want a higher quality copy, sorry pal - you’re stuck with 1 mbps in 2024, even though a 50 mbps copy was available on usenet 14 years ago.

<span id="hardware-format-restrictions"></span>
== Hardware &amp; Format Restrictions ==

<span id="bluray-limitations"></span>
=== Bluray Limitations ===

Let’s not even get started on the limitations regarding 4k blurays. If you want to rip a 4k bluray, you can’t - you’re stuck at 1080p unless you buy a drive like the Pioneer BDR-2213 with older firmware that allows you to back up a copy of what you '''legally bought &amp; paid for.'''

<span id="digital-books-another-broken-promise"></span>
=== Digital Books: Another Broken Promise ===

<span id="kindle-purchases-and-country-restrictions"></span>
==== '''Kindle Purchases and Country Restrictions''' ====

Amazon Kindle thinks that [https://x.com/krishnanrohit/status/1772011384206672370 moving to a new country means you should lose all your Kindle books]. Imagine paying to buy a book and then having it disappear as your flight leaves your country’s border. Welcome to 2024.

It gets better. Amazon has instructions on their website to ''“transfer”'' your account, but their own customer service reps are clueless on how any of it works.

Amazon hides behind licensing agreements and geo-restrictions to justify this anti-ownership garbage. While you’re given the '''impression''' you’re ''“purchasing”'' a book, you’re actually just getting a temporary license tied to the country you bought it in. Move countries? f you, buy the book again.

This isn’t about technical limitations. This is about '''control'''. Amazon and companies like it are obsessed with locking down what you own. They know you won’t read the fine print until you’re angry, but by then, it’s too late.

This isn’t just about Kindle. It’s about digital purchases everywhere. You don’t actually own what you buy. Whether it’s Kindle books, movies on Amazon, or games on Sony, the story is the same: they sell you the illusion of ownership &amp; lock you down with restriction after restriction after they’ve pocketed your money. You should consider yourself lucky if they even allow you to keep using what you bought in a restricted manner; sometimes they just take it away &amp; leave you nothing at all.

<span id="the-broken-system-of-consumer-protection"></span>

== The Broken System of Consumer Protection ==

<span id="no-real-consequences"></span>
=== No Real Consequences ===

In the United States, consumer protection &amp; our congress/senate no longer create laws that protect the rights of consumers. Everything I described above is disgustingly unethical; if I advertised as deceptively as Sony, Netflix, Disney, or Discovery did, I would be fined out of business if my customers hadn’t ransacked my store &amp; broken the window. But they get away with it.

Technically; legally, these companies are in the right for what they’re doing. and even if they weren’t, when they do something horribly illegal &amp; unethical, what our joke of a government [https://docs.fcc.gov/public/attachments/FCC-24-40A1.pdf fines them] is amounts to [https://investors.att.com/~/media/Files/A/ATT-IR-V2/t-2023-12-31-10k-2024.pdf 0.37% of their net profit for the year]

Piracy is how you take back ownership when the government that exists to [https://legaljournal.princeton.edu/the-supreme-courts-perversion-of-property-rights/?utm_source=chatgpt.com protect your property rights] takes 37% of your paycheck &amp; [https://imgur.com/a/cPTYhYh allows your $3600 television to roofie you &amp; steal your personal data] &amp; your content providers to [https://docs.fcc.gov/public/attachments/FCC-24-40A2.pdf sell your location to bounty hunters] or [https://web.archive.org/web/20240820100244/https://www.nytimes.com/2024/08/14/nyregion/disney-wrongful-death-lawsuit-arbitration.html kill your wife &amp; get away with it] due to forced arbitration agreement in a video streaming app terms of service. Forced arbitration agreements like the one that Disney tried to use to justify shielding themselves from any liability for a person’s death are ''still legal in America today''.

<span id="a-personal-note-supporting-content-while-rejecting-control"></span>

== A Personal Note: Supporting Content While Rejecting Control ==

<span id="i-pay-for-content-you-should-too."></span>
=== I pay for content; you should too. ===

I buy &amp; pay for what I find valuable. Whether it’s my bluray copy of [https://www.blu-ray.com/Tori-Amos-Live-At-Montreux-1991-1992/19345/ Tori Amos’ Live at Montreux] concert from 1992 or my 22 year old copy of SuSE Linux Professional 8.1 I bought at Best Buy for $79.99. Not only do I pay for 32 year old concerts, I pay for software you can ''legally download for free'' if I think it’s worthwhile. For all the trouble I give open source software, I paid for a copy of GNU/Linux back when you had to [https://www.linuxquestions.org/questions/linux-newbie-8/k3b-scsi-emulation-127823/ compile your own kernel to burn a CD].

<gallery mode="packed-hover" heights=250 widths=400 perrow=2>
File:suse.jpeg
</gallery>

I believe in paying for what I find valuable. It empowers me to ask for what I am worth when I provide value to others. I believe in fair exchange of value.

That being said: I will '''never''' let someone else tell me what I '''CAN''' or '''CANNOT''' do with what I ''bought and paid for''; nor will I ever tolerate being provided a ''worse experience as a paying customer'' than what I get as a non-paying customer. The limitations placed on your experience when you buy media are not worse due to scarcity or technological limitations; rather, the technological limitations are PURPOSELY PUT IN PLACE BY THE PERSON YOU ARE PAYING.

At my business, we see our customers as partners; not adversaries. If a business I am seeking a service or good from treats me like the enemy after I’ve given them money; I will treat them in kind.

'''Piracy is how you retain control over what you bought and paid for. Never feel guilty about that.''' But remember that it is on us to pay for what we find valuable, &amp; demonstrate that we are willing to pay for what we find valuable, if we want to live in a world where non-abusive business models win.

<span id="a-nuanced-view-of-digital-rights-piracy"></span>

== A Nuanced View of Digital Rights &amp; Piracy ==

Not all situations where customers choose piracy are equal. Here’s a hierarchy of scenarios that I’ve ordered from most to least “justifiable” or “ok” to make the point. When you hear the words ''“piracy”'' or ''“copyright infringement”'', '''they are often used to paint anyone who does not accept being bent over by companies that wish to re-define what it means to ''“own”'' something; if not take away the concept of ownership completely.''' Do not accept the premise of assholes, or laws, that pretend that each of the following scenarios below are the same.

<span id="legitimate-ownership-issues"></span>
=== Legitimate Ownership Issues ===

<span id="degraded-physical-media---no-replacement-available"></span>
==== 1. Degraded Physical Media - No Replacement Available ====

You paid for physical media that has degraded. The content is no longer for sale anywhere, and you need a way to access what you rightfully &amp; legally purchased. It is still protected by copyright, but you are literally incapable of purchasing it again due to lack of availability that is not your fault.

<span id="degraded-physical-media---replacement-available"></span>
==== 2. Degraded Physical Media - Replacement Available ====

You paid for physical media that has degraded. While you could buy it again, you’ve already paid the creators once for lifetime access.

<span id="lost-digital-purchase---no-repurchase-option"></span>
==== 3. Lost Digital Purchase - No Repurchase Option ====

You purchased digital media that was accidentally erased/lost, and it’s no longer available for sale anywhere.

<span id="lost-digital-purchase---repurchase-available"></span>
==== 4. Lost Digital Purchase - Repurchase Available ====

You purchased digital media that was accidentally erased/lost. While it’s still for sale, you’ve already paid once for what was advertised as a “purchase.”

<span id="corporate-deception-control"></span>
=== Corporate Deception &amp; Control ===

<span id="the-purchase-that-wasnt"></span>
==== 5. The “Purchase” That Wasn’t ====

You '''PURCHASED''' digital media; using the commonly understood definition of PURCHASE that existed from the 14th century that 99% of customers understand - permanent ownership. but it stopped working because someone you never met decided ''“fuck you”''.

<span id="bait-switch-streaming"></span>
==== 6. Bait &amp; Switch Streaming ====

You paid for a streaming service specifically advertised with certain content. That content was removed with no refund option, &amp; now requires a second subscription to a different service to access.

<span id="rental-vs.-purchase-confusion"></span>
==== 7. Rental vs. Purchase Confusion ====

The distinction between rental &amp; purchase was unclear or deliberately obscured so you’d think you were '''PURCHASING''' something.

<span id="technical-restrictions-quality-issues"></span>

=== Technical Restrictions &amp; Quality Issues ===

<span id="the-4k-lockout"></span>
==== 8. The 4K Lockout ====

You paid for higher quality content (like 4K) but received lower quality (720p/1080p) or same resolution but with radically reduced, horrible bitrate, due to artificial DRM restrictions that were buried at the end of a bs 30 page EULA; if made available to you at all. Your hardware is fully capable, but artificial limitations put in place by the content distributor keep you from using what you bought &amp; paid for.

<span id="drm-workarounds-while-supporting-creators"></span>
==== 9. DRM Workarounds While Supporting Creators ====

You purchase physical media to support creators but use a pirated copy to avoid DRM restrictions or long shipping delays.

<span id="random-shitty-scenarios"></span>
=== Random Shitty Scenarios ===

<span id="region-lock-issues"></span>
==== 10. Region Lock Issues ====

Content is completely unavailable in your region with no legal purchase option, even though you’re willing to pay.

<span id="drm-protest-without-support"></span>
==== 11. DRM Protest Without Support ====

You reject DRM-restricted content but also choose not to purchase available DRM-free options when they exist, turning a blind eye.

<span id="selective-support"></span>
==== 12. Selective Support ====

You support creators directly but won’t acknowledge how distributors &amp; other parts of the content creation pipeline process add value (paying for studios, people who support the recording &amp; making of content, etc.)

<span id="indefensible-positions"></span>

=== Indefensible Positions ===

<span id="false-justification"></span>
==== 13. False Justification ====

Using DRM and middlemen as excuses while never actually supporting creators in any way.

<span id="empty-protests"></span>
==== 14. Empty Protests ====

Claiming DRM opposition while pirating even when DRM-free options exist.

<span id="simply-being-an-asshole"></span>
==== 15. Simply being an asshole ====

Taking content with no intention to ever support creators; even the ones you truly enjoy, even when you have the money to pay for it, while using a litany of excuses to justify the behavior.

<span id="i-just-want-free-stuff"></span>
==== 16. "I just want free stuff." ====

No justification, no excuse, no attempt to support creators—just pure entitlement.

<span id="conclusion"></span>

== Conclusion ==

While many of these scenarios are brought about via legit grievances with the current state of you-own-nothing-digital-media with spyware on top, in my opinion, there’s a clear ethical distinction between retaining access to content you’ve purchased versus never intending to support people who have provided you value. The higher items on this list represent what I find to be genuine consumer rights abuses, while the lower items represent entitled cunts hiding behind moral superiority who lack the honesty to say they just don’t want to pay for anything. Even an ''“I hate that industry and want to bleed them dry &amp; don’t care about the consequences”'' would be more acceptable to me, for at least it’s honest.

When I advocate for having full control over what you buy &amp; pay for, I’m specifically addressing the upper scenarios where customers have made good-faith attempts to support creators but are getting screwed left &amp; right by content companies &amp; distributors by artificial restrictions, deceptive practices, and technical limitations.

<span id="final-thoughts"></span>

= Final Thoughts =

The joy of this process is in making it your own. I gave you a rough outline here of what you can do; a guide that shows you what is possible, so that you could have those little kicks of dopamine that show up when something works. Those kicks of dopamine are imperative to you feeling good &amp; moving forward. Without them, most people give up &amp; stop trying. If you stop trying, you never learn.

The purpose of this guide wasn’t to tell you this is the only way to do all of these things. Rather, it was to provide you a framework that I ''100% know works since I followed it myself''. I’ve already set up a system like this, one chunk at a time, over 14 years. I can tell you what to do, but putting together instructions that actually work is only possible if I run through the process in realtime to ensure everything I am telling ''you'' to do actually works for me. By going through the guide as I write it, if I leave something out of this guide - then what I am doing won’t work.

My hope is that once you are un-encumbered by the Linux-isms &amp; open source-isms &amp; ''RTFM'' elitist forum assholes that link you to documentation that is wrong or makes no sense, that you’ll feel empowered to make something that kicks ass on your own. This is not the ''only'' way to do this, nor is this even the ''“right”'' way. There’s no such thing as the ''“right”'' way (although there are many '''WRONG''' ways!).

You don’t have to clone this setup. Figure out what works for you, build something cool in small pieces &amp; baby steps. You don’t have to do it all at once. Enjoy the journey! I can’t wait to see what you build. That’s it for today, &amp; as always, '''I hope you learned something!'''