# Proxmox Management

This repo contains script used to manage a multi-tenant proxmox environment for the [Reudnetz w.V.]().
These scripts were created beacuse the ansible modules lacked the necessary features.

---

This hacky solution will be replaced by a clean terraform abstraction !

---

## Overview

We use users, groups, resource pools, and roles to build a multi-tenante proxmox instance.

Proxmox by design is not a "cloud native" hypervisor platform. Although it is possible to seperate users and resources from on another there is no way give users/organisations a quota of cpu,memory,disk from which they can assign virtual machines/containers to themselfs.

To overcome this limitation an admin creates those vms/containers and then attaches them read-only to the specific user/organisation.


### Moving Parts

For our hypervisor setup we need to administrate two components:

* the hypervisor platform
* routing/switching

Currently there is no idempotent automation for the hypervisor platform. State changes can be done via scripts or the cli directly (or via the WebGUI of course).

The routing/switching side is fully automated. State changes are configured by modifying the `vlan_vms.yml` file.

### Terminology

Our abstraction is based on these objects:

* users
* organisations
* vms

Users can be part of multiple organisations.
Every vm is bound to a single organisation.

### Proxmox Implementation

In proxmox we use a few more components to implement our abstraction:

* users
* groups (used as organisations)
* resource pools
	* used to map permissions for groups
* acls


Under the (proxmox) hood our users (as in from our abstraction) are normal users.
Organisations are abstracted via groups.

VMs and file-based storage are assigned to so called `Resource pools`.

We now map permissions for a specific organisation (group) to a specific resource pool. This seperates the resources and creates a multi-tenant platform.

There are  WIP


## Requirements

HOW TO USE THESE SCRIPTS IN AN PVE ENVIRONMENT.
TBD

## Create an Organisation

1. create proxmox related objects
```
./create_organisation ${name}
```



## Create a VM

**prerequisites**:
* there should be a organisation created for that vm
* you should have a list of `authorized_keys` for that organisation saved as a file on that node

0. search for next useable vlan/vm-id in `vlan_vms.yml`
  * either use an old but freed id (block should be commented)
  * or use next free one
1. check that we have the newest images
```
grep debian download_images
ls /root/images
# execute ./download_images if script url does not match images directory
```

2. create vm
```
./create_vm ${vmid} ${orga}-{vmname}-${vmid} ${orga}
./get_linklocal_for_vm ${vmid}
```

3. add vlan configuration to `vlan_vms.yml` and deploy ansible
```
vim vlan_vms.yml
[...] copy block or create new one - change prefixed accordingly
[...] enter ll for v6 prefix
# deploy routers/switches
ansible-playbook -CD cores.yml -l cores,core_switches
ansible-playbook -D cores.yml -l cores,core_switches
# deploy hyperjump
ansible-playbook -CD vms.yml -l hyperjump
ansible-playbook -D vms.yml -l hyperjump
```

4. configure cloud-init settings for vm
```
qm set ${vmid} --ipconfig0 "ip=${v4-cidr},gw=${v4-gw},ip6=${v6-prefix},gw6=fe80::1"
qm set ${vmid} --sshkeys ${path_to_authorized_keys}
```

5. change vm specifications to order
```
# increase memory / cpu
qm set ${vmid} -memory 8192 --cores 4

# add more storage
#  use customer-disks on hyper01 and hyper03
#  use customer-disks-slow on hyper02 (dedicated sata ssd pool with more space but less bandwith)
qm set ${vmid} --virtio1 customer-disks-slow:250,iothread=1,discard=on
```

6. start vm
```
qm start ${vmid}
```

7. check reachability of vm
```
ping6 ${v6prefix}
nc ${v6prefix} 22
```

8. notify customer

```
vmid: ${vmid}
vmname: ${vmname}
user: debian
authentication: key based auth only

v4: ${v4-cidr}
v4gw: ${v4-gw}
v6: ${v6-prefix}
v6gw: fe80::1

v6 reachable from everywhere - v4 only inside the reudnetz
ssh reverse proxy (for v4 access only) available via: ssh -p 2${vmid} <user>@hyperjump.reudnetz.org

best effort service - expect downtimes - backup your stuff !
```

## Destroy a VM

commands need to be executed on node where vm resides

1. schedule removal of replicated storage (if configured)
```
pvesr list | grep ${vmid}
pvesr delete ${vmid}-${replication_id}
echo wait till removal is complete - check with pvesr list
```
2. remove vm
```
qm shutdown ${vmid}
qm destroy --purge 1 ${vmid}
```
3. remove vms from `vlan_vms.yml` and deploy
```
vim vlan_vms.yml
[...]
# remove router and switch config
ansible-playbook -CD cores.yml -l cores,core_switches
ansible-playbook -D cores.yml -l cores,core_switches
# remove ssh reverse proxy config
ansible-playbook -CD vms.yml -l hyperjump
ansible-playbook -D vms.yml -l hyperjump
```

## Destroy User

```
./delete_user ${username}@pve
```

## Destroy organisation

1. remove orga and associated objects from proxmox cluster
```
./delete_organisation ${org}
```
2. destroy remaining zfs volumes on other nodes
```
# execute on all pve nodes
zfs destroy -r rpool/customer/${orga}-images
```

## Add/Remove user from organisation

1. check which groups the user currently belongs to
```
pveum group list | grep ${user}@pve
```
2. change groups memberships
```
# add to group
pveum user modify ${user}@pve --groups ${currentgroups},${new_group}
# remove from group
pveum user modify ${user}@pve --groups ${groups-without-other-group}
```