%
% Proxmox.tex
%
% Fork Sand IT Manual
%
% Copyright (C) 2018, Fork Sand, Inc.
% Copyright (C) 2017, Jeff Moe
% Copyright (C) 2017 Aleph Objects, Inc.
%
% This document is licensed under the Creative Commons Attribution 4.0
% International Public License (CC BY-SA 4.0) by Fork Sand, Inc.
%
% XXX TODO: Proxmox-GUI-login.png 

\section{Overview}
Proxmox is a virtual machine manager. 

The private cloud deployment will be based on Proxmox version 5.x.
%There are only Debian 8 (Jessie, oldstable) releases.
%Debian hasn't packaged Proxmox since wheezy. It has it in sid, but even
%that is an old version. The only packages available for Debian are the
%upstream ones for Jessie made by Proxmox.
UPDATE: although Proxmox isn't listed on Debian 9 (Stretch) packages,
there is an installation manual for 5.x version, which is great.

Documentation:
\url{https://pve.proxmox.com/wiki/Documentation}

\begin{figure}[!htb]
\includegraphics[keepaspectratio=true,height=1.10\textheight,width=1.00\textwidth,angle=0]{www-proxmox.png}
 \caption{Proxmox Website}
 \label{fig:www-proxmox}
\end{figure}

\begin{itemize}
 \item Website: \\ \url{https://proxmox.com/}
 \item Debian Stretch ISO: \\
   \url{http://download.proxmox.com/iso/proxmox-ve_5.1-3.iso}

includes the following:

$\cdot$ Complete operating system (Debian Linux, 64-bit)

$\cdot$ The Proxmox VE installer, which partitions the hard
drive(s) with ext4, ext3, xfs or ZFS and installs the
operating system.

$\cdot$ Proxmox VE kernel (Linux) with LXC and KVM support
Proxmox VE Administration Guide 9/309

$\cdot$ Complete toolset for administering virtual machines,
containers and all necessary resources

$\cdot$ Web based management interface for using the toolset

 \item Debian Stretch admin guide: \\
   \url{https://pve.proxmox.com/pve-docs/pve-admin-guide.pdf}
\end{itemize}


The following servers will be deployed to host Proxmox and the KVMs:

\begin{itemize}
 %\item \texttt{waz-kvm-001.forksand.com}
 %\item \texttt{waz-kvm-002.forksand.com} --- Virtual Machine Server 2
 %\item \texttt{waz-kvm-003.forksand.com} --- Virtual Machine Server 3
 %\item \texttt{waz-kvm-004.forksand.com} --- Virtual Machine Server 4
 %\item \texttt{waz-kvm-005.forksand.com} --- Virtual Machine Server 5
 %\item \texttt{waz-sun-001.forksand.com} --- Proxmox Web GUI 1
 %\item \texttt{waz-sun-002.forksand.com} --- Proxmox Web GUI 2
 %\item \texttt{forksand-hk1} --- Virtual Machine Node 1
 %\item \texttt{forksand-hk2} --- Virtual Machine Node 2
 %\item \texttt{forksand-hk3} --- Virtual Machine Node 3
 \item \texttt{forksand-shark1} --- Virtual Machine Node 1
 \item \texttt{forksand-shark2} --- Virtual Machine Node 2
 \item \texttt{forksand-shark3} --- Virtual Machine Node 3
 \item \texttt{forksand-shark4} --- Virtual Machine Node 4
 \item \texttt{forksand-the} --- Virtual Machine Testing Node 1
 \item \texttt{forksand-truck} --- Virtual Machine Testing Node 2
\end{itemize}

%\subsection{Virtual Machine Servers}
%KVM virtual machine servers. Fast CPU, with lots of RAM. Uses Ceph to store
%virtual images.
%
%\subsection{Proxmox Web GUI Servers}
%A Proxmox's Web GUI for administration of the \gls{cluster}.

\subsection{Virtual Machine Nodes}
Virtual machine nodes. Fast CPU, with lots of RAM. Uses Ceph to store
virtual images.

Every node includes a Proxmox's Web GUI service for administration of the \gls{cluster}.
Any nodes included into the \gls{cluster} may be configured by requesting to any node's GUI.

\begin{figure}[!htb]
\includegraphics[keepaspectratio=true,height=1.10\textheight,width=1.00\textwidth,angle=0]{proxmox-gui.png}
 \caption{Proxmox Sunstone Web Admin GUI}
 \label{fig:proxmox-gui}
\end{figure}


\begin{minted}{sh}
echo "deb http://download.proxmox.com/debian/pve stretch pve-no-subscription" \
> /etc/apt/sources.list.d/pve-install-repo.list
wget http://download.proxmox.com/debian/proxmox-ve-release-5.x.gpg \
-O /etc/apt/trusted.gpg.d/proxmox-ve-release-5.x.gpg
apt-get update
apt-get -y dist-upgrade --download-only
DEBIAN_FRONTEND=noninteractive apt-get -y \
-o Dpkg::Options::="--force-confdef" \
-o Dpkg::Options::="--force-confnew" dist-upgrade
apt-get -y install ksm-control-daemon proxmox-veupdate-grub
apt remove os-prober
\end{minted}

\section{Bugs}
Things that are bugs or at least aren't configured correctly.

\section{GUI Configuration}
At this point, you should have the Proxmox server up and running.

\textcolor[rgb]{0.80,0.00,0.00}{
Todo check related, modify/replace unrelated
}

%See scripts in \texttt{source/resources/ns24} for automation.
\begin{enumerate}
\item Set up Linux Bridge (see fig. \ref{fig:shark2linuxbridge} p.\pageref{fig:shark2linuxbridge}):
 %Code: \\ \texttt{ssh -N -C -L 9869:localhost:9869 ns24}
\item In workstation, open browser to urls: \\
URL: \url{http://localhost:8001/}, for shark1 \\
URL: \url{http://localhost:8002/}, for shark2 \\
URL: \url{http://localhost:8003/}, for shark3 \\
URL: \url{http://localhost:8004/}, for shark4 \\
See example at fig. \ref{fig:proxmox-gui-port}:
\begin{figure}[!htb]
    \includegraphics[keepaspectratio=true,height=1.10\textheight,width=1.00\textwidth,angle=0]{shark2/23.png}
    \label{fig:proxmox-gui-port}
    \caption{Browse shark2 node, visible port No.}
\end{figure}
Info: This goes through https with self-signed certificate.
\item \texttt{Hostname} Changing the hostname and IP is not possible after
    \gls{cluster} creation. Unlike OpenNebula.
%\item Click \texttt{Infrastructure}.
%\item Click \texttt{Hosts}.
%\item Click The \texttt{+} plus icon.
%\item Enter the hostname of the KVM server you want to use, such as the Sunstone server itself.
% \texttt{Type: KVM}
% \texttt{Hostname: ns24}
%\item Click \texttt{Create}.
%\item Repeatedly hit the reload button that's the two arrows in a circle, as it goes thru
% stages of setup, starting at \texttt{INIT}.
\item Confirm status is \texttt{ON}.
\end{enumerate}

\section{GUI Deploy Image}
\textcolor[rgb]{0.80,0.00,0.00}{Todo check related, modify/replace unrelated}
\begin{minted}{sh}
This is a quick and dirty way to deploy a first test image.
NOTE: It is note privacy aware, as it pulls the image from the
Proxmox ``store''.
\end{minted}

\textcolor[rgb]{0.80,0.00,0.00}{Todo check related, modify/replace unrelated}
\begin{minted}{sh}
How to deploy an image from the Proxmox App store:
\begin{enumerate}
 \item Click \texttt{Storage}.
 \item Click \texttt{Apps}
 \item Click \texttt{Debian 9 - KVM}.
 \item Click on the icon that is a cloud with an arrow in it. This downloads it to Proxmox.
 \item Select a datastore by clicking the \texttt{default} line.
 \item Leave name and all that the same, and click \texttt{Download}.
 \item Click \texttt{Images} under \texttt{Storage} in the left column.
 \item Hit the refresh icon repeatedly.
 \item When \texttt{Status} is \texttt{READY}, it is good to go.
 \item Click \texttt{Templates} in the left column.
 \item Click \texttt{VMs}.
 \item Click \texttt{Debian 9 - KVM}.
 \item Click \texttt{Instantiate}.
 \item \texttt{VM Name} enter \texttt{deb9}.
 \item \texttt{Number of instances} enter \texttt{1}.
 \item \texttt{Memory} enter \texttt{768}.
 \item \texttt{CPU} enter \texttt{1}.
 \item Click the slider to \texttt{Instantiate as persistent}.
 \item Click \texttt{Instantiate}.
 \item Click \texttt{Instances} in the left column.
 \item Click \texttt{VMs}.
 \item Click the reload icon, repeat.
 \item It is good when \texttt{Status} is \texttt{RUNNING}.
 \item Set up an \texttt{ssh} tunnel so VNC can be used:
  \texttt{ssh -N -C -L 29876:localhost:29876 ns24}
 \item Click on the little monitor icon to launch VNC.
 \item Look at booted up screen at \texttt{login:} prompt.
 \item This means a Debian KVM booted up and the VNC is working.
  There is no password for the \texttt{root} account, only \texttt{ssh} is available.
  So without network setup, you can't really do anything with this image.
  Booted, it just shows it works.
\end{enumerate}

I think delete this section, it would go before the \texttt{Templates} above.:
\begin{enumerate}
 \item Click \texttt{Debian 9 - KVM}.
 \item PROBABLY NO: Click \texttt{Clone} to make a local copy.
 \item PROBABLY NO: It will say \texttt{Copy of Debian 9 - KVM}, leave as-is, click \texttt{Clone}.
 \item Click on the icon with three dots.
 \item Click \texttt{Make Persistent}.
 \item Click on the icon with three dots.
 \item Click \texttt{Enable}.
\end{enumerate}
\end{minted}

\section{Proxmox Networking}
Create --> Linux Bridge: vmbr0

XXX best way for this server? No subnet.

\textcolor[rgb]{0.80,0.00,0.00}{taken from forksand-shark4-bootstrap}

\begin{minted}{sh}
source /etc/network/interfaces.d/*
auto enp1s0f1
iface enp1s0f1 inet static
        address 70.39.103.210/29
        gateway 70.39.103.209
        dns-nameservers 208.67.222.222
        dns-search forksand.com
\end{minted}

As user \texttt{jebba}, on the server, run this to generate a key.
Then paste that key into Sunstone under "SSH Public Key".

\begin{minted}{sh}
ssh-keygen -t ed25519
\end{minted}

\begin{minted}{sh}
# XXX test. Use this IP and interface, so no 192.168.0.0 but real IPs.
# Comment this out:
auto eth0:27
iface eth0:27 inet static
address 174.128.229.158
netmask 255.255.255.224
gateway 174.128.229.129
\end{minted}

XXX Check if IP forwarding is needed in \texttt{/etc/sysctl.conf}.

If things are set up to use a bridge and 192.168.100.100,
\texttt{iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE}
Will bring things up to NAT.


XXX The port forwarding is forwarding all port 53 to guest at the moment.