|
|
%
|
|
|
% Ansible.tex
|
|
|
%
|
|
|
% Fork Sand IT Manual
|
|
|
%
|
|
|
% Copyright (C) 2018, Fork Sand, Inc.
|
|
|
% Copyright (C) 2017, Jeff Moe
|
|
|
%
|
|
|
% This document is licensed under the Creative Commons Attribution 4.0
|
|
|
% International Public License (CC BY-SA 4.0) by Fork Sand, Inc.
|
|
|
%
|
|
|
\section{Ansible Cloud Management}
|
|
|
Use \texttt{ansible} for management of servers.
|
|
|
|
|
|
\begin{itemize}
|
|
|
\item Ansible ---
|
|
|
Website: \\ \url{https://ansible.com}
|
|
|
\item Ansible Github ---
|
|
|
Repo: \\ \url{https://github.com/ansible/ansible.git}
|
|
|
\item DebOps: \url{https://docs.debops.org/en/latest/index.html}
|
|
|
\end{itemize}
|
|
|
|
|
|
\subsection{Glossary}
|
|
|
An \texttt{Ansible playbook} is an organized unit of scripts that defines
|
|
|
work for a server configuration managed by the automation tool \texttt{ansible}.
|
|
|
|
|
|
\subsection{Build Ansible Debian Package}\label{ssec:bansdpac}
|
|
|
The version of Ansible is 2.2 in Debian 9 (stable/Stretch).
|
|
|
The current stable release is Ansible 2.4.
|
|
|
Below documents how to build a 2.4 package for Debian 9.
|
|
|
|
|
|
\begin{minted}{sh}
|
|
|
# To build a Debian package:
|
|
|
# Check here for latest version:
|
|
|
# https://packages.debian.org/sid/ansible
|
|
|
wget http://http.debian.net/debian/pool/main/a/ansible/ansible_2.4.0.0+dfsg-1.debian.tar.xz
|
|
|
wget http://http.debian.net/debian/pool/main/a/ansible/ansible_2.4.0.0+dfsg.orig.tar.gz
|
|
|
|
|
|
# Install some deps
|
|
|
apt-get install debhelper python-all python-crypto python-setuptools python-yaml asciidoc python-nose python-passlib dh-python
|
|
|
|
|
|
tar xf ansible_2.4.0.0+dfsg.orig.tar.gz
|
|
|
cd ansible-2.4.0.0/
|
|
|
tar xf ../ansible_2.4.0.0+dfsg-1.debian.tar.xz
|
|
|
|
|
|
# Update version:
|
|
|
echo -n " -- Jeff Moe <moe@blagblagblag.org> " ; date "+%a, %d %b %Y %H:%M:%S %z"
|
|
|
vim debian/changelog
|
|
|
|
|
|
dpkg-buildpackage -rfakeroot -S -uc -us -sa
|
|
|
dpkg-buildpackage -rfakeroot -b -uc
|
|
|
|
|
|
# That will produce this file to be installed:
|
|
|
dpkg -i ansible_2.4.0.0+dfsg-2_all.deb
|
|
|
apt-get -f install
|
|
|
|
|
|
# If you want the Ansible git archive:
|
|
|
git clone https://github.com/ansible/ansible.git --recursive
|
|
|
\end{minted}
|
|
|
|
|
|
\subsection{Ansible Initial Configuration}
|
|
|
Here is how to set up Ansible after initially installing it.
|
|
|
This is run on the system adminstrator's workstation.
|
|
|
|
|
|
\begin{minted}{sh}
|
|
|
# Quick and dirty test by setting up a host and running `uptime`.
|
|
|
mkdir -p ~/.ansible
|
|
|
echo ns1 > ~/.ansible/hosts
|
|
|
ansible -i ~/.ansible/hosts ns1 -a 'uptime'
|
|
|
\end{minted}
|
|
|
|
|
|
~/.ansible.cfg
|
|
|
\begin{minted}{sh}
|
|
|
[defaults]
|
|
|
inventory = $HOME/.ansible/hosts
|
|
|
|
|
|
[ssh_connection]
|
|
|
ssh_args = -o ControlMaster=auto -o ControlPersist=300s
|
|
|
pipelining = True
|
|
|
\end{minted}
|
|
|
|
|
|
To generate a full list of \texttt{ns} hosts, run the script:
|
|
|
|
|
|
\begin{minted}{sh}
|
|
|
cd source/resources/servers
|
|
|
echo "[ns]" > ~/.ansible/hosts
|
|
|
./ns-serverlist-ansible.sh >> ~/.ansible/hosts
|
|
|
ansible -i ~/.ansible/hosts ns -a "uptime"
|
|
|
# Find the failed hosts and remove them from ~/.ansible/hosts.
|
|
|
echo "[ns]" > ~/.ansible/hosts.tmp
|
|
|
ansible -f 32 -i ~/.ansible/hosts ns -a "echo" | grep ^ns | grep SUCCESS | cut -f 1 -d " " | sort -V >> ~/.ansible/hosts.tmp
|
|
|
mv ~/.ansible/hosts ~/.ansible/hosts.old
|
|
|
mv ~/.ansible/hosts.tmp ~/.ansible/hosts
|
|
|
# Test it works:
|
|
|
ansible -i ~/.ansible/hosts ns -a "uptime"
|
|
|
# To get a ton of info about each host:
|
|
|
ansible -i ~/.ansible/hosts ns -m setup
|
|
|
\end{minted}
|
|
|
|
|
|
XXX Fix, make sure everyone has \texttt{/usr/bin/python} available for \texttt{ansible}:
|
|
|
|
|
|
\begin{minted}{sh}
|
|
|
# XXXX SOME HOSTS DON'T HAVE /usr/bin/python
|
|
|
# JUST PYTHON3. HOSTS THAT DIDN'T HAVE /usr/bin/python:
|
|
|
# (ALL OVH)
|
|
|
# ns14 ns15 ns21 ns22
|
|
|
apt install python python-minimal
|
|
|
\end{minted}
|
|
|
|
|
|
Set up some playbooks, grab examples:
|
|
|
|
|
|
\begin{minted}{sh}
|
|
|
git clone https://github.com/ansible/ansible-examples.git
|
|
|
cd ansible-examples/
|
|
|
\end{minted}
|
|
|
|
|
|
\section{Ansible Debian Mail}
|
|
|
Assuming \texttt{ansible} is built according to section \ref{ssec:bansdpac}
|
|
|
on p.\pageref{ssec:bansdpac}, the following requirements are met.
|
|
|
|
|
|
\subsection{Requirements}
|
|
|
The following applications are required to utilize this this section objectives.
|
|
|
Ansible can be installed using Python PIP.
|
|
|
\begin{itemize}
|
|
|
\item \texttt{Ansible} 2.4.x+
|
|
|
\item \texttt{Python} 2.7.x+
|
|
|
\textcolor[rgb]{0.80,0.00,0.00}{Todo clarify confusion over version requirements}
|
|
|
\end{itemize}
|
|
|
|
|
|
\subsection{Quick Start}
|
|
|
The following steps will help quickly set up and execute this section objectives.
|
|
|
|
|
|
\texttt{Project Configuration}
|
|
|
|
|
|
The following files need to be edited and configured before executing this playbook.
|
|
|
|
|
|
\begin{table}[!htb]
|
|
|
\caption{Files to be edited} % \label{tab:tech}
|
|
|
\begin{tabular}{|l|l|}
|
|
|
\hline
|
|
|
\multicolumn {1}{|l|}{ File}&
|
|
|
\multicolumn {1}{l|}{ Description} \\ \hline
|
|
|
groups\char`_vars/all.yml & Server credential information and domain variables \\ \hline
|
|
|
inventory.yml & List of server IPs to connect to \\ \hline
|
|
|
\end{tabular}
|
|
|
\end{table}
|
|
|
|
|
|
\texttt{Playbook Execution}
|
|
|
|
|
|
After having configured the server credentials and added the server IP to the inventory, use the following command to execute the playbook.
|
|
|
|
|
|
\begin{minted}{sh}
|
|
|
ansible-playbook -i inventory.yml site.yml -vvv
|
|
|
\end{minted}
|
|
|
|
|
|
\subsection{Project Structure}
|
|
|
The following tree depicts the high level structure of this Ansible project.
|
|
|
|
|
|
\begin{minted}{sh}
|
|
|
|-- ansible.cfg
|
|
|
|-- group_vars
|
|
|
|<7C><> -- all.yml
|
|
|
|-- inventory.yml
|
|
|
|-- LICENSE.AGPLv3
|
|
|
|-- LICENSE.GPLv3
|
|
|
|-- README.md
|
|
|
|-- roles
|
|
|
|<7C><> |-- dkim_configuration
|
|
|
|<7C><> |-- dovecot_configuration
|
|
|
|<7C><> |-- fail2ban_configuration
|
|
|
|<7C><> |-- letsencrypt_configuration
|
|
|
|<7C><> |-- mikegleasonjr.firewall
|
|
|
|<7C><> |-- outputs
|
|
|
|<7C><> |-- postfix_configuration
|
|
|
|<7C><> |-- server_tasks
|
|
|
|<7C><> |-- spamassassin_configuration
|
|
|
|<7C><> -- sqlgrey_configuration
|
|
|
|-- playbook_execution.log
|
|
|
-- site.yml
|
|
|
\end{minted}
|
|
|
|
|
|
\texttt{File and Directory Descriptions}
|
|
|
|
|
|
The following table consists of a description of what each file and directory stands for.
|
|
|
|
|
|
\begin{table}[!htb]
|
|
|
\caption{File and Directory Descriptions} % \label{tab:tech}
|
|
|
\begin{tabular}{|l|l|}
|
|
|
\hline
|
|
|
\multicolumn {1}{|l|}{ Name}&
|
|
|
\multicolumn {1}{l|}{ Description} \\ \hline
|
|
|
site.yml & Master playbook. Executes all roles in sequential order \\ \hline
|
|
|
inventory.yml & Inventory file containing server IP addresses \\ \hline
|
|
|
ansible.cfg & Ansible configuration file for various Ansible options. \\ \hline
|
|
|
group\char`_vars/ & Group\char`_vars directory contains variable files for the entire group. \\
|
|
|
& The files are named according to the group name. 'all.yml' = group 'all' \\ \hline
|
|
|
group\char`_vars/all.yml & Group variables for the 'all' group. Contains server connection \\
|
|
|
& information along with domain variables \\ \hline
|
|
|
roles/ & Directory containing all roles needed by this project \\ \hline
|
|
|
\end{tabular}
|
|
|
\end{table}
|
|
|
|
|
|
\texttt{Role descriptions}
|
|
|
|
|
|
The following table consists of descriptions of each role and their purpose. The roles listed below are listed in the required order of execution to ensure successful completion of the playbook.
|
|
|
|
|
|
\begin{table}[!htb]
|
|
|
\caption{Role descriptions} % \label{tab:tech}
|
|
|
\begin{tabular}{|l|l|l|}
|
|
|
\hline
|
|
|
\multicolumn {1}{|l|}{Role Name}&
|
|
|
\multicolumn {1}{|l|}{Role Description}&
|
|
|
\multicolumn {1}{l|}{Depends on} \\ \hline
|
|
|
server\char`_tasks & This roles performs all server tasks. Updating & mikebleasonjr.firewall \\
|
|
|
& server, configuring SSH, disable IPv6, etc. & \\
|
|
|
& Depends on the mikegleasonjr.firewall role. & \\ \hline
|
|
|
mikegleasonjr.firewall & This role set up iptables rules. It is called & None \\
|
|
|
& and ran by the server\char`_tasks roles. & \\ \hline
|
|
|
letsencrypt\char`_configuration & This role installs and executes let's encrypt & None \\ \hline
|
|
|
postfix\char`_configuration & This roles installs postfix, configures postfix & letsencrypt\char`_configuration \\
|
|
|
& using postconf, and sets up virtual file, & \\
|
|
|
& master.cf file, and aliases file & \\ \hline
|
|
|
dkim\char`_configuration & This roles installs OpenDKIM, OpenDMARC & None \\
|
|
|
& and configures them. & \\ \hline
|
|
|
dovecot\char`_configuration & This role installs and configures dovecot & letsencrypt\char`_configuration \\ \hline
|
|
|
spamassassin\char`_configuration & This role installs spamassassin. & None \\ \hline
|
|
|
sqlgrey\char`_configuration & This role installs sqlgrey. & None \\ \hline
|
|
|
fail2ban\char`_configuration & This role installs fail2ban. & None \\ \hline
|
|
|
outputs & This role gathers DNS information for the & None \\
|
|
|
& SPF, DMARC, and DKIM records and & \\
|
|
|
& outputs them to the screen. & \\ \hline
|
|
|
\end{tabular}
|
|
|
\end{table}
|
|
|
|
|
|
\subsection{Ansible Logging}
|
|
|
Execution processes described in this section are automatically logged to a file called `playbook-execution.log` in the root directory of the project. The path to this log file can be changed by editing `ansible.cfg` in the project root directory and specifying a different path.
|
|
|
|
|
|
\subsection{Troubleshooting}
|
|
|
Ansible has a built in debug output. Simply run Ansible with a `-v`. There are 5 levels of debug output and they are denoted by the number of v's listed. Each level up provide more debug output than the level before it.
|
|
|
|
|
|
\begin{minted}{sh}
|
|
|
Level 1: `-v`
|
|
|
|
|
|
Level 2: `-vv`
|
|
|
|
|
|
Level 3: `-vvv`
|
|
|
|
|
|
Level 4: `-vvvv`
|
|
|
|
|
|
Level 5: `-vvvvv`
|
|
|
\end{minted}
|
|
|
|
|
|
Example execution with level 3 debug output:
|
|
|
|
|
|
\begin{minted}{sh}
|
|
|
ansible-playbook -i inventory.yml site.yml -vvv
|
|
|
\end{minted}
|
|
|
|
|
|
\section{Ansible Gitea}
|
|
|
Assuming \texttt{ansible} is built according to section \ref{ssec:bansdpac}
|
|
|
on p.\pageref{ssec:bansdpac}, the following requirements are met.
|
|
|
|
|
|
\subsection{Requirements}
|
|
|
The following applications are required to utilize this this section objectives.
|
|
|
Ansible can be installed using Python PIP.
|
|
|
\begin{itemize}
|
|
|
\item \texttt{Ansible} 2.4.x+
|
|
|
\item \texttt{Python} 2.7.x
|
|
|
\end{itemize}
|
|
|
|
|
|
\subsection{Quick Start}
|
|
|
The following steps will help quickly set up and execute this section objectives.
|
|
|
|
|
|
\texttt{Project Configuration}
|
|
|
|
|
|
The following files need to be edited and configured before executing this playbook.
|
|
|
|
|
|
\begin{table}[!htb]
|
|
|
\caption{Files to be edited} % \label{tab:tech}
|
|
|
\begin{tabular}{|l|l|}
|
|
|
\hline
|
|
|
\multicolumn {1}{|l|}{ File}&
|
|
|
\multicolumn {1}{l|}{ Description} \\ \hline
|
|
|
roles/gitea/default/main.yml & Variables for Gitea configuration \\
|
|
|
& (default - standalone with sqlite) \\ \hline
|
|
|
roles/nginx/default/main.yml & Variables for Nginx and Letsencrypt configuration \\ \hline
|
|
|
inventory.yml & List of server IPs to connect to \\ \hline
|
|
|
\end{tabular}
|
|
|
\end{table}
|
|
|
|
|
|
\qquad \\
|
|
|
\texttt{Playbook Execution}
|
|
|
|
|
|
After having configured the server credentials and added the server IP to the inventory, use the following command to execute the playbook.
|
|
|
|
|
|
\begin{minted}{sh}
|
|
|
ansible-playbook -i inventory.yml site.yml -vvv
|
|
|
\end{minted}
|
|
|
|
|
|
\subsection{Project Structure}
|
|
|
The following tree depicts the high level structure of this Ansible project.
|
|
|
|
|
|
\begin{minted}{sh}
|
|
|
|-- inventory.yml
|
|
|
|-- LICENSE.AGPLv3
|
|
|
|-- LICENSE.GPLv3
|
|
|
|-- README.md
|
|
|
|-- roles
|
|
|
|<7C><> |-- gitea
|
|
|
|<7C><> -- nginx
|
|
|
|-- playbook_execution.log
|
|
|
-- site.yml
|
|
|
\end{minted}
|
|
|
|
|
|
\texttt{File and Directory Descriptions}
|
|
|
|
|
|
The following table consists of a description of what each file and directory stands for.
|
|
|
|
|
|
\begin{table}[!htb]
|
|
|
\caption{File and Directory Descriptions} % \label{tab:tech}
|
|
|
\begin{tabular}{|l|l|}
|
|
|
\hline
|
|
|
\multicolumn {1}{|l|}{ Name}&
|
|
|
\multicolumn {1}{l|}{ Description} \\ \hline
|
|
|
site.yml & Master playbook. Executes all roles in sequential order \\ \hline
|
|
|
inventory.yml & Inventory file containing server IP addresses \\ \hline
|
|
|
ansible.cfg & Ansible configuration file for various Ansible options. \\ \hline
|
|
|
roles/ & Directory containing all roles needed by this project \\ \hline
|
|
|
\end{tabular}
|
|
|
\end{table}
|
|
|
|
|
|
\qquad \\
|
|
|
|
|
|
\texttt{Role descriptions}
|
|
|
|
|
|
The following table consists of descriptions of each role and their purpose. The roles listed below are listed in the required order of execution to ensure successful completion of the playbook.
|
|
|
|
|
|
\begin{table}[!htb]
|
|
|
\caption{Role descriptions} % \label{tab:tech}
|
|
|
\begin{tabular}{|l|l|l|}
|
|
|
\hline
|
|
|
\multicolumn {1}{|l|}{Role Name}&
|
|
|
\multicolumn {1}{|l|}{Role Description} \\ \hline
|
|
|
gitea & This roles performs installation and configuration of Gitea server \\ \hline
|
|
|
nginx & This roles performs installation and configuration of Nginx server \\ \hline
|
|
|
\end{tabular}
|
|
|
\end{table}
|
|
|
|
|
|
\texttt{Role parameters}
|
|
|
|
|
|
\qquad \\
|
|
|
|
|
|
\texttt{\qquad Gitea role parameters}
|
|
|
\begin{minted}{sh}
|
|
|
# Application name
|
|
|
gitea_app_name: "Gitea"
|
|
|
# Application gitea_user_repo_limit
|
|
|
gitea_user: "gitea"
|
|
|
# Application home
|
|
|
gitea_home: "/var/lib/gitea"
|
|
|
# Repo Limit
|
|
|
gitea_user_repo_limit: -1
|
|
|
# Domain Name (FOR REVER PROXY LEAVE AS DEFAULT)
|
|
|
gitea_http_domain: localhost
|
|
|
# Gitea url (FOR REVER PROXY LEAVE AS DEFAULT)
|
|
|
gitea_root_url: http://localhost:3000
|
|
|
# Protocol (FOR REVER PROXY LEAVE AS DEFAULT)
|
|
|
gitea_protocol: http
|
|
|
# listen IP (FOR REVER PROXY LEAVE AS DEFAULT)
|
|
|
gitea_http_listen: 127.0.0.1
|
|
|
# Listen port (FOR REVER PROXY LEAVE AS DEFAULT)
|
|
|
gitea_http_port: 3000
|
|
|
# HTTP git Options
|
|
|
gitea_disable_http_git: false
|
|
|
# Offline mode options
|
|
|
gitea_offline_mode: true
|
|
|
\end{minted}
|
|
|
|
|
|
\qquad \qquad DB details
|
|
|
|
|
|
\begin{minted}{sh}
|
|
|
# DB Type 'mysql', 'postgres' or 'sqlite3'
|
|
|
gitea_db_type: sqlite3
|
|
|
# DB host
|
|
|
gitea_db_host: 127.0.0.0:3306
|
|
|
# DB name
|
|
|
gitea_db_name: root
|
|
|
# DB username
|
|
|
gitea_db_user: gitea
|
|
|
# DB password
|
|
|
gitea_db_passord: lel
|
|
|
# DB ssl options
|
|
|
gitea_db_ssl: disable
|
|
|
# DB path (Not needed for postgres and mysql hash it in template file )
|
|
|
gitea_db_path: "{{ gitea_home }}/data/gitea.db"
|
|
|
\end{minted}
|
|
|
|
|
|
\qquad \qquad SSH Details
|
|
|
|
|
|
\begin{minted}{sh}
|
|
|
# SSH Listen IP
|
|
|
gitea_ssh_listen: 0.0.0.0
|
|
|
# SSH domain
|
|
|
gitea_ssh_domain: localhost
|
|
|
# SSH options
|
|
|
gitea_start_ssh: true
|
|
|
# SSH post
|
|
|
gitea_ssh_port: 2222
|
|
|
|
|
|
\qquad \qquad Gitea secret key
|
|
|
|
|
|
\begin{minted}{sh}
|
|
|
# gitea key (GENERATE A NEW KEY)
|
|
|
gitea_secret_key: T0pS3cr31
|
|
|
\end{minted}
|
|
|
|
|
|
\qquad \qquad General Settings
|
|
|
|
|
|
\begin{minted}{sh}
|
|
|
# User email settings
|
|
|
gitea_show_user_email: false
|
|
|
# User avatar settings
|
|
|
gitea_disable_gravatar: true
|
|
|
# User register options
|
|
|
gitea_disable_registration: false
|
|
|
# User signup options
|
|
|
gitea_require_signin: true
|
|
|
# User captcha options
|
|
|
gitea_enable_captcha: true
|
|
|
\end{minted}
|
|
|
|
|
|
\qquad \\
|
|
|
|
|
|
\texttt{\qquad Nginx roles parameters}
|
|
|
\begin{minted}{sh}
|
|
|
# Domain name for the server
|
|
|
nginx_domain_name: "test.hostnats.com"
|
|
|
# Gitea listening port
|
|
|
gitea_http_port: 3000
|
|
|
# letsencrypt email address
|
|
|
letsencrypt_email: "test@example.com"
|
|
|
\end{minted}
|
|
|
|
|
|
\subsection{Ansible Logging}
|
|
|
Execution processes described in this section are automatically logged to a file called `playbook-execution.log` in the root directory of the project. The path to this log file can be changed by editing `ansible.cfg` in the project root directory and specifying a different path.
|
|
|
|
|
|
\subsection{Troubleshooting}
|
|
|
Ansible has a built in debug output. Simply run Ansible with a `-v`. There are 5 levels of debug output and they are denoted by the number of v's listed. Each level up provide more debug output than the level before it.
|
|
|
|
|
|
\begin{minted}{sh}
|
|
|
Level 1: `-v`
|
|
|
|
|
|
Level 2: `-vv`
|
|
|
|
|
|
Level 3: `-vvv`
|
|
|
|
|
|
Level 4: `-vvvv`
|
|
|
|
|
|
Level 5: `-vvvvv`
|
|
|
\end{minted}
|
|
|
|
|
|
Example execution with level 3 debug output:
|
|
|
|
|
|
\begin{minted}{sh}
|
|
|
ansible-playbook -i inventory.yml site.yml -vvv
|
|
|
\end{minted}
|
|
|
|