% % Ansible.tex % % Fork Sand IT Manual % % Copyright (C) 2018, Fork Sand, Inc. % Copyright (C) 2017, Jeff Moe % % This document is licensed under the Creative Commons Attribution 4.0 % International Public License (CC BY-SA 4.0) by Fork Sand, Inc. % \section{Ansible Cloud Management} Use \texttt{ansible} for management of servers. \begin{itemize} \item Ansible --- Website: \\ \url{https://ansible.com} \item Ansible Github --- Repo: \\ \url{https://github.com/ansible/ansible.git} \item DebOps: \url{https://docs.debops.org/en/latest/index.html} \end{itemize} \subsection{Glossary} An \texttt{Ansible playbook} is an organized unit of scripts that defines work for a server configuration managed by the automation tool \texttt{ansible}. \subsection{Build Ansible Debian Package}\label{ssec:bansdpac} The version of Ansible is 2.2 in Debian 9 (stable/Stretch). The current stable release is Ansible 2.4. Below documents how to build a 2.4 package for Debian 9. \begin{minted}{sh} # To build a Debian package: # Check here for latest version: # https://packages.debian.org/sid/ansible wget http://http.debian.net/debian/pool/main/a/ansible/ansible_2.4.0.0+dfsg-1.debian.tar.xz wget http://http.debian.net/debian/pool/main/a/ansible/ansible_2.4.0.0+dfsg.orig.tar.gz # Install some deps apt-get install debhelper python-all python-crypto python-setuptools python-yaml asciidoc python-nose python-passlib dh-python tar xf ansible_2.4.0.0+dfsg.orig.tar.gz cd ansible-2.4.0.0/ tar xf ../ansible_2.4.0.0+dfsg-1.debian.tar.xz # Update version: echo -n " -- Jeff Moe " ; date "+%a, %d %b %Y %H:%M:%S %z" vim debian/changelog dpkg-buildpackage -rfakeroot -S -uc -us -sa dpkg-buildpackage -rfakeroot -b -uc # That will produce this file to be installed: dpkg -i ansible_2.4.0.0+dfsg-2_all.deb apt-get -f install # If you want the Ansible git archive: git clone https://github.com/ansible/ansible.git --recursive \end{minted} \subsection{Ansible Initial Configuration} Here is how to set up Ansible after initially installing it. This is run on the system adminstrator's workstation. \begin{minted}{sh} # Quick and dirty test by setting up a host and running `uptime`. mkdir -p ~/.ansible echo ns1 > ~/.ansible/hosts ansible -i ~/.ansible/hosts ns1 -a 'uptime' \end{minted} ~/.ansible.cfg \begin{minted}{sh} [defaults] inventory = $HOME/.ansible/hosts [ssh_connection] ssh_args = -o ControlMaster=auto -o ControlPersist=300s pipelining = True \end{minted} To generate a full list of \texttt{ns} hosts, run the script: \begin{minted}{sh} cd source/resources/servers echo "[ns]" > ~/.ansible/hosts ./ns-serverlist-ansible.sh >> ~/.ansible/hosts ansible -i ~/.ansible/hosts ns -a "uptime" # Find the failed hosts and remove them from ~/.ansible/hosts. echo "[ns]" > ~/.ansible/hosts.tmp ansible -f 32 -i ~/.ansible/hosts ns -a "echo" | grep ^ns | grep SUCCESS | cut -f 1 -d " " | sort -V >> ~/.ansible/hosts.tmp mv ~/.ansible/hosts ~/.ansible/hosts.old mv ~/.ansible/hosts.tmp ~/.ansible/hosts # Test it works: ansible -i ~/.ansible/hosts ns -a "uptime" # To get a ton of info about each host: ansible -i ~/.ansible/hosts ns -m setup \end{minted} XXX Fix, make sure everyone has \texttt{/usr/bin/python} available for \texttt{ansible}: \begin{minted}{sh} # XXXX SOME HOSTS DON'T HAVE /usr/bin/python # JUST PYTHON3. HOSTS THAT DIDN'T HAVE /usr/bin/python: # (ALL OVH) # ns14 ns15 ns21 ns22 apt install python python-minimal \end{minted} Set up some playbooks, grab examples: \begin{minted}{sh} git clone https://github.com/ansible/ansible-examples.git cd ansible-examples/ \end{minted} \section{Ansible Debian Mail} Assuming \texttt{ansible} is built according to section \ref{ssec:bansdpac} on p.\pageref{ssec:bansdpac}, the following requirements are met. \subsection{Requirements} The following applications are required to utilize this this section objectives. Ansible can be installed using Python PIP. \begin{itemize} \item \texttt{Ansible} 2.4.x+ \item \texttt{Python} 2.7.x+ \textcolor[rgb]{0.80,0.00,0.00}{Todo clarify confusion over version requirements} \end{itemize} \subsection{Quick Start} The following steps will help quickly set up and execute this section objectives. \texttt{Project Configuration} The following files need to be edited and configured before executing this playbook. \begin{table}[!htb] \caption{Files to be edited} % \label{tab:tech} \begin{tabular}{|l|l|} \hline \multicolumn {1}{|l|}{ File}& \multicolumn {1}{l|}{ Description} \\ \hline groups\char`_vars/all.yml & Server credential information and domain variables \\ \hline inventory.yml & List of server IPs to connect to \\ \hline \end{tabular} \end{table} \texttt{Playbook Execution} After having configured the server credentials and added the server IP to the inventory, use the following command to execute the playbook. \begin{minted}{sh} ansible-playbook -i inventory.yml site.yml -vvv \end{minted} \subsection{Project Structure} The following tree depicts the high level structure of this Ansible project. \begin{minted}{sh} |-- ansible.cfg |-- group_vars |   -- all.yml |-- inventory.yml |-- LICENSE.AGPLv3 |-- LICENSE.GPLv3 |-- README.md |-- roles |   |-- dkim_configuration |   |-- dovecot_configuration |   |-- fail2ban_configuration |   |-- letsencrypt_configuration |   |-- mikegleasonjr.firewall |   |-- outputs |   |-- postfix_configuration |   |-- server_tasks |   |-- spamassassin_configuration |   -- sqlgrey_configuration |-- playbook_execution.log -- site.yml \end{minted} \texttt{File and Directory Descriptions} The following table consists of a description of what each file and directory stands for. \begin{table}[!htb] \caption{File and Directory Descriptions} % \label{tab:tech} \begin{tabular}{|l|l|} \hline \multicolumn {1}{|l|}{ Name}& \multicolumn {1}{l|}{ Description} \\ \hline site.yml & Master playbook. Executes all roles in sequential order \\ \hline inventory.yml & Inventory file containing server IP addresses \\ \hline ansible.cfg & Ansible configuration file for various Ansible options. \\ \hline group\char`_vars/ & Group\char`_vars directory contains variable files for the entire group. \\ & The files are named according to the group name. 'all.yml' = group 'all' \\ \hline group\char`_vars/all.yml & Group variables for the 'all' group. Contains server connection \\ & information along with domain variables \\ \hline roles/ & Directory containing all roles needed by this project \\ \hline \end{tabular} \end{table} \texttt{Role descriptions} The following table consists of descriptions of each role and their purpose. The roles listed below are listed in the required order of execution to ensure successful completion of the playbook. \begin{table}[!htb] \caption{Role descriptions} % \label{tab:tech} \begin{tabular}{|l|l|l|} \hline \multicolumn {1}{|l|}{Role Name}& \multicolumn {1}{|l|}{Role Description}& \multicolumn {1}{l|}{Depends on} \\ \hline server\char`_tasks & This roles performs all server tasks. Updating & mikebleasonjr.firewall \\ & server, configuring SSH, disable IPv6, etc. & \\ & Depends on the mikegleasonjr.firewall role. & \\ \hline mikegleasonjr.firewall & This role set up iptables rules. It is called & None \\ & and ran by the server\char`_tasks roles. & \\ \hline letsencrypt\char`_configuration & This role installs and executes let's encrypt & None \\ \hline postfix\char`_configuration & This roles installs postfix, configures postfix & letsencrypt\char`_configuration \\ & using postconf, and sets up virtual file, & \\ & master.cf file, and aliases file & \\ \hline dkim\char`_configuration & This roles installs OpenDKIM, OpenDMARC & None \\ & and configures them. & \\ \hline dovecot\char`_configuration & This role installs and configures dovecot & letsencrypt\char`_configuration \\ \hline spamassassin\char`_configuration & This role installs spamassassin. & None \\ \hline sqlgrey\char`_configuration & This role installs sqlgrey. & None \\ \hline fail2ban\char`_configuration & This role installs fail2ban. & None \\ \hline outputs & This role gathers DNS information for the & None \\ & SPF, DMARC, and DKIM records and & \\ & outputs them to the screen. & \\ \hline \end{tabular} \end{table} \subsection{Ansible Logging} Execution processes described in this section are automatically logged to a file called `playbook-execution.log` in the root directory of the project. The path to this log file can be changed by editing `ansible.cfg` in the project root directory and specifying a different path. \subsection{Troubleshooting} Ansible has a built in debug output. Simply run Ansible with a `-v`. There are 5 levels of debug output and they are denoted by the number of v's listed. Each level up provide more debug output than the level before it. \begin{minted}{sh} Level 1: `-v` Level 2: `-vv` Level 3: `-vvv` Level 4: `-vvvv` Level 5: `-vvvvv` \end{minted} Example execution with level 3 debug output: \begin{minted}{sh} ansible-playbook -i inventory.yml site.yml -vvv \end{minted} \section{Ansible Gitea} Assuming \texttt{ansible} is built according to section \ref{ssec:bansdpac} on p.\pageref{ssec:bansdpac}, the following requirements are met. \subsection{Requirements} The following applications are required to utilize this this section objectives. Ansible can be installed using Python PIP. \begin{itemize} \item \texttt{Ansible} 2.4.x+ \item \texttt{Python} 2.7.x \end{itemize} \subsection{Quick Start} The following steps will help quickly set up and execute this section objectives. \texttt{Project Configuration} The following files need to be edited and configured before executing this playbook. \begin{table}[!htb] \caption{Files to be edited} % \label{tab:tech} \begin{tabular}{|l|l|} \hline \multicolumn {1}{|l|}{ File}& \multicolumn {1}{l|}{ Description} \\ \hline roles/gitea/default/main.yml & Variables for Gitea configuration \\ & (default - standalone with sqlite) \\ \hline roles/nginx/default/main.yml & Variables for Nginx and Letsencrypt configuration \\ \hline inventory.yml & List of server IPs to connect to \\ \hline \end{tabular} \end{table} \qquad \\ \texttt{Playbook Execution} After having configured the server credentials and added the server IP to the inventory, use the following command to execute the playbook. \begin{minted}{sh} ansible-playbook -i inventory.yml site.yml -vvv \end{minted} \subsection{Project Structure} The following tree depicts the high level structure of this Ansible project. \begin{minted}{sh} |-- inventory.yml |-- LICENSE.AGPLv3 |-- LICENSE.GPLv3 |-- README.md |-- roles |   |-- gitea |   -- nginx |-- playbook_execution.log -- site.yml \end{minted} \texttt{File and Directory Descriptions} The following table consists of a description of what each file and directory stands for. \begin{table}[!htb] \caption{File and Directory Descriptions} % \label{tab:tech} \begin{tabular}{|l|l|} \hline \multicolumn {1}{|l|}{ Name}& \multicolumn {1}{l|}{ Description} \\ \hline site.yml & Master playbook. Executes all roles in sequential order \\ \hline inventory.yml & Inventory file containing server IP addresses \\ \hline ansible.cfg & Ansible configuration file for various Ansible options. \\ \hline roles/ & Directory containing all roles needed by this project \\ \hline \end{tabular} \end{table} \qquad \\ \texttt{Role descriptions} The following table consists of descriptions of each role and their purpose. The roles listed below are listed in the required order of execution to ensure successful completion of the playbook. \begin{table}[!htb] \caption{Role descriptions} % \label{tab:tech} \begin{tabular}{|l|l|l|} \hline \multicolumn {1}{|l|}{Role Name}& \multicolumn {1}{|l|}{Role Description} \\ \hline gitea & This roles performs installation and configuration of Gitea server \\ \hline nginx & This roles performs installation and configuration of Nginx server \\ \hline \end{tabular} \end{table} \texttt{Role parameters} \qquad \\ \texttt{\qquad Gitea role parameters} \begin{minted}{sh} # Application name gitea_app_name: "Gitea" # Application gitea_user_repo_limit gitea_user: "gitea" # Application home gitea_home: "/var/lib/gitea" # Repo Limit gitea_user_repo_limit: -1 # Domain Name (FOR REVER PROXY LEAVE AS DEFAULT) gitea_http_domain: localhost # Gitea url (FOR REVER PROXY LEAVE AS DEFAULT) gitea_root_url: http://localhost:3000 # Protocol (FOR REVER PROXY LEAVE AS DEFAULT) gitea_protocol: http # listen IP (FOR REVER PROXY LEAVE AS DEFAULT) gitea_http_listen: 127.0.0.1 # Listen port (FOR REVER PROXY LEAVE AS DEFAULT) gitea_http_port: 3000 # HTTP git Options gitea_disable_http_git: false # Offline mode options gitea_offline_mode: true \end{minted} \qquad \qquad DB details \begin{minted}{sh} # DB Type 'mysql', 'postgres' or 'sqlite3' gitea_db_type: sqlite3 # DB host gitea_db_host: 127.0.0.0:3306 # DB name gitea_db_name: root # DB username gitea_db_user: gitea # DB password gitea_db_passord: lel # DB ssl options gitea_db_ssl: disable # DB path (Not needed for postgres and mysql hash it in template file ) gitea_db_path: "{{ gitea_home }}/data/gitea.db" \end{minted} \qquad \qquad SSH Details \begin{minted}{sh} # SSH Listen IP gitea_ssh_listen: 0.0.0.0 # SSH domain gitea_ssh_domain: localhost # SSH options gitea_start_ssh: true # SSH post gitea_ssh_port: 2222 \qquad \qquad Gitea secret key \begin{minted}{sh} # gitea key (GENERATE A NEW KEY) gitea_secret_key: T0pS3cr31 \end{minted} \qquad \qquad General Settings \begin{minted}{sh} # User email settings gitea_show_user_email: false # User avatar settings gitea_disable_gravatar: true # User register options gitea_disable_registration: false # User signup options gitea_require_signin: true # User captcha options gitea_enable_captcha: true \end{minted} \qquad \\ \texttt{\qquad Nginx roles parameters} \begin{minted}{sh} # Domain name for the server nginx_domain_name: "test.hostnats.com" # Gitea listening port gitea_http_port: 3000 # letsencrypt email address letsencrypt_email: "test@example.com" \end{minted} \subsection{Ansible Logging} Execution processes described in this section are automatically logged to a file called `playbook-execution.log` in the root directory of the project. The path to this log file can be changed by editing `ansible.cfg` in the project root directory and specifying a different path. \subsection{Troubleshooting} Ansible has a built in debug output. Simply run Ansible with a `-v`. There are 5 levels of debug output and they are denoted by the number of v's listed. Each level up provide more debug output than the level before it. \begin{minted}{sh} Level 1: `-v` Level 2: `-vv` Level 3: `-vvv` Level 4: `-vvvv` Level 5: `-vvvvv` \end{minted} Example execution with level 3 debug output: \begin{minted}{sh} ansible-playbook -i inventory.yml site.yml -vvv \end{minted}