My Homelab Infrastructure: Deploying a Hashicorp Stack with Ansible
After years of tinkering with various homelab setups, I’ve finally settled on what I consider to be the perfect infrastructure stack for my needs. In this post, I’ll walk you through how I use Ansible to deploy and manage a complete Hashicorp stack (Nomad, Consul, Vault) along with Caddy as a reverse proxy, Consul-Template for dynamic configuration, and Cloudflared for secure tunneling.
Note: I’ve open-sourced all the configuration files and deployment scripts on GitHub. If you find this guide helpful, consider supporting my work on Patreon.
Why This Stack?
Before diving into the technical details, let me explain why I chose this particular combination:
- Nomad: Lightweight and easy to use scheduler that doesn’t require Kubernetes complexity. Perfect for a homelab where you want container orchestration without the operational overhead.
- Consul: Service discovery, configuration, and segmentation mesh. The glue that holds everything together, providing DNS, health checks, and service mesh capabilities.
- Vault: Secrets management and data protection. Keeps my API keys, certificates, and other sensitive information secure.
- Caddy: Modern, automatic HTTPS web server and reverse proxy. So much easier to configure than Nginx or Apache, with automatic certificate management.
- Consul-Template: Dynamic configuration generation from Consul data. This is the magic that makes everything adapt automatically.
- Cloudflared: Secure tunneling without exposing ports to the internet. No more port forwarding or VPNs needed to access my services remotely.
After looking into Kubernetes, I found it was overkill for my needs. The Nomad ecosystem provides most of the same benefits with a fraction of the complexity. It’s the perfect middle ground between running containers manually and a full-blown Kubernetes cluster.
Infrastructure Overview
My setup consists of several types of nodes:
- Server nodes: These run Nomad, Consul, and Vault servers. I have three of these for high availability, with one running on my main server and two on Synology NAS devices using VMs.
- Client nodes: These run Nomad, Consul, and Vault clients that execute workloads. I have a mix of physical machines and VMs, some running on Synology NAS devices, others on dedicated hardware.
- Edge nodes: These handle ingress traffic with Caddy and Cloudflared. I have one dedicated edge node on a Raspberry Pi that acts as the entry point for all external traffic.
This separation of concerns allows me to scale each component independently. If I need more compute capacity, I can add more client nodes. If I need more ingress capacity, I can add more edge nodes.
Ansible Structure
The entire deployment is managed through Ansible, with a structure that separates bootstrap operations from day-to-day management:
.
├── ansible.cfg # Main Ansible configuration
├── bootstrap/ # Initial setup files
│ ├── ansible.cfg # Bootstrap-specific Ansible config
│ ├── boot.yml # Initial bootstrap playbook
│ ├── files/ # Template files
│ ├── group_vars/ # Group variables
│ ├── requirements.yml # Required Ansible roles
│ ├── roles/ # Custom roles
│ └── site.yml # Main deployment playbook
├── hosts.yml # Inventory file
└── Makefile # Convenience commands
Setting Up the Environment
The first step in deploying this stack is to install the required Ansible roles. Here’s what my requirements.yml looks like:
- name: setup-nomad
src: https://github.com/username/ansible-nomad.git
- name: setup-consul
src: https://github.com/ansible-community/ansible-consul.git
- name: setup-vault
src: https://github.com/ansible-community/ansible-vault.git
- name: setup-caddy
src: https://github.com/caddy-ansible/caddy-ansible
- name: setup-consul-template
src: https://github.com/griggheo/ansible-consul-template
- name: setup-cloudflared
src: https://github.com/papanito/ansible-role-cloudflared
I install these roles with a simple command:
ansible-galaxy install -r bootstrap/requirements.yml
Inventory Configuration
The inventory file (hosts.yml) is where I define all the nodes in my infrastructure. Here’s a simplified example:
all:
children:
edge_nodes:
hosts:
edge01:
network_address: "10.0.1.5"
ansible_host: "10.0.1.5"
consul_node_role: server
network_interface: eth0
node_kind: metal
nomad_cluster:
hosts:
server01:
network_address: "10.0.1.10"
ansible_host: "10.0.1.10"
nomad_node_role: both
consul_node_role: server
network_interface: eth0
node_kind: metal
Each node has specific attributes that determine its role and configuration. This detailed inventory allows Ansible to configure each node appropriately based on its role and characteristics.
Configuration Variables
One of the most powerful aspects of Ansible is its variable system. I use group variables to define configuration for different types of nodes. Here are some key configurations:
Consul Configuration
# From group_vars/all.yml
consul_version: 1.16.0
consul_connect_enabled: true
consul_datacenter: "homelab"
consul_domain: "homelab"
consul_bootstrap_expect: true
consul_bootstrap_expect_value: 3
consul_dnsmasq_enable: true
consul_config_custom:
addresses:
grpc: "{{ network_address }}"
ports:
grpc: 8502
connect:
enabled: true
Nomad Configuration
# From group_vars/nomad_cluster.yml
nomad_version: 1.6.0
nomad_datacenter: "homelab"
nomad_region: "us-east-1"
nomad_bootstrap_expect: 3
nomad_telemetry: true
nomad_telemetry_prometheus_metrics: true
nomad_plugins:
raw_exec:
config:
enabled: true
docker:
config:
volumes:
enabled: true
allow_caps: ["NET_ADMIN", "CAP_CHOWN", "SYS_ADMIN"]
allow_privileged: true
Edge Node Configuration
# From group_vars/edge_nodes.yml
cf_tunnels:
homelab-tunnel:
routes:
dns:
- "{{ inventory_hostname }}"
- "*.homelab.local"
ingress:
- hostname: "{{ inventory_hostname }}.homelab.local"
service: "https://{{ network_address }}:443"
- hostname: "homelab.local"
service: "https://{{ network_address }}:443"
- hostname: "*.homelab.local"
service: "https://{{ network_address }}:443"
- service: http_status:404
consul_template_templates:
- name: "caddy-template.ctmpl"
dest: "/etc/caddy/Caddyfile"
cmd: "systemctl reload caddy.service"
Dynamic Caddy Configuration with Consul-Template
One of the coolest parts of my setup is using Consul-Template to dynamically generate Caddy configurations based on services registered in Consul. Here’s a simplified version of my template:
{{ range services }}
{{ range service .Name }}
{{ if (.Tags | contains "caddy") }}
{{ scratch.MapSetX "vhosts" .Name true }}
{{ if .Tags | contains "public" }}
{{ scratch.MapSet "vhosts" .Name false }}
{{ end }}
{{ end }}
{{ end }}
{{ end }}
{
http_port 80
https_port 443
acme_dns cloudflare YOUR_CLOUDFLARE_TOKEN
storage "consul" {
address "{{ network_address }}:8500"
prefix "caddytls"
}
}
https://.homelab.local {
{{ range $vhost, $private := scratch.Get "vhosts" }}
@{{ $vhost }} host {{ $vhost }}.homelab.local
handle @{{ $vhost }} {
{{ if $private }}
@blocked not remote_ip 10.0.1.0/24
respond @blocked "Access denied" 403
{{ end }}
{{ range services }}
{{ range service .Name }}
{{ if (and (.Tags | contains "caddy") (eq .Name $vhost)) }}
reverse_proxy http://{{ .Address }}:{{ .Port }} {
header_up Host {http.request.host}
}
{{ end }}
{{ end }}
{{ end }}
}
{{ end }}
tls {
dns cloudflare YOUR_CLOUDFLARE_TOKEN
}
}
This template automatically creates virtual hosts for any service in Consul tagged with “caddy”, and can restrict access to private services. When a new service is registered in Consul with the appropriate tags, Consul-Template regenerates the Caddyfile and reloads Caddy, making the service available via HTTPS without any manual intervention.
Deployment Process
The deployment is handled by the main playbook in bootstrap/site.yml. Here’s a simplified version:
---
- hosts: 'nomad_cluster'
roles:
- role: setup-docker
become: true
when: ansible_distribution == 'Debian' or ansible_distribution == 'Ubuntu'
tags:
- docker
- role: setup-nomad
become: true
tags:
- nomad
- role: setup-consul
become: true
tags:
- consul
- role: setup-vault
become: true
tags:
- vault
- hosts: 'edge_nodes'
roles:
- role: setup-consul
become: true
tags:
- consul
- role: setup-cloudflared
become: true
tags:
- cloudflared
- role: setup-consul-template
become: true
tags:
- consul-template
- role: setup-caddy
become: true
tags:
- caddy
I can run the entire deployment at once or use tags to target specific components:
# Full deployment
ansible-playbook -i hosts.yml bootstrap/site.yml
# Just update Caddy configuration
ansible-playbook -i hosts.yml bootstrap/site.yml --tags caddy
Running Services with Nomad
Once the infrastructure is set up, I can deploy services using Nomad job files. Here’s a simple example of a job file for running a web application:
job "webapp" {
datacenters = ["homelab"]
type = "service"
group "app" {
count = 2
network {
port "http" {
to = 8080
}
}
service {
name = "webapp"
port = "http"
tags = ["caddy", "public"]
check {
type = "http"
path = "/health"
interval = "10s"
timeout = "2s"
}
connect {
sidecar_service {}
}
}
task "server" {
driver = "docker"
config {
image = "mywebapp:latest"
ports = ["http"]
}
vault {
policies = ["webapp"]
}
env {
PORT = "${NOMAD_PORT_http}"
}
template {
data = <<EOH
{{ with secret "kv/data/webapp" }}
DB_PASSWORD={{ .Data.data.DB_PASSWORD }}
{{ end }}
EOH
destination = "secrets/vault.env"
env = true
}
resources {
cpu = 500
memory = 256
}
}
}
}
This job deploys two instances of a web application, registers it with Consul, and makes it available via HTTPS through Caddy. It also fetches a database password from Vault and injects it as an environment variable.
Securing the Setup
Security is a major concern for any homelab, especially one that’s accessible from the internet. My setup includes several layers of security:
- Vault for secrets management, ensuring sensitive information is encrypted at rest and in transit
- Consul Connect for service mesh with mTLS, providing secure service-to-service communication
- Cloudflared for secure tunneling without exposing ports, eliminating the attack surface of open ports
- Caddy with automatic HTTPS, ensuring all traffic is encrypted
- ACLs enabled on both Nomad and Consul, providing fine-grained access control
Real-World Usage
This setup has been running my homelab for over a year now, and it’s been incredibly reliable. I run a variety of services, including:
- Home automation with Home Assistant and Node-RED
- Personal websites and blogs
- Development environments for various projects
- Monitoring and logging with Prometheus, Grafana, and Loki
The beauty of this setup is that I can deploy new services with minimal effort. I just create a Nomad job file, deploy it, and everything else happens automatically.
Lessons Learned
Building this infrastructure wasn’t without challenges. Here are some lessons I learned along the way:
- Start simple: I initially tried to do too much at once. It’s better to start with a basic setup and add complexity as needed.
- Automate everything: Any manual step will eventually cause problems. Automation is key to reliability.
- Document everything: Even with automation, documentation is crucial for understanding why things are set up a certain way.
- Test in isolation: Before deploying changes to production, test them in a separate environment.
- Backup regularly: Even with high availability, backups are essential for disaster recovery.
Conclusion
This setup has been rock solid for me. The combination of Nomad, Consul, Vault, Caddy, and Cloudflared gives me a powerful, flexible, and secure platform for running all my homelab services.
The best part is that it’s all automated with Ansible, so I can rebuild or expand my infrastructure with minimal effort. If a node fails, I can replace it and have it back in the cluster in minutes.
If you’re looking to build a similar setup, I hope this post gives you a good starting point. You can find all the configuration files and deployment scripts in my GitHub repository. If you find this guide helpful, consider supporting my work on Patreon.
Remember, the goal of a homelab is to learn and have fun. Don’t be afraid to experiment and make mistakes. That’s how we learn!