From 1fd57fe774888d95799f174efca7a06d426804b8 Mon Sep 17 00:00:00 2001 From: Gregor Michels Date: Wed, 7 Jan 2026 14:32:52 +0100 Subject: [PATCH] WIP: add README.md --- README.md | 219 ++++++++++++++++++++++++++++++++++++++++++++++++------ 1 file changed, 197 insertions(+), 22 deletions(-) diff --git a/README.md b/README.md index 81da068..d3d9cfe 100644 --- a/README.md +++ b/README.md @@ -1,42 +1,217 @@ # Proxmox Management This repo contains script used to manage a multi-tenant proxmox environment for the [Reudnetz w.V.](). -These scripts were created because ansible currently lacks modules to configure users and groups. +These scripts were created beacuse the ansible modules lacked the necessary features. -**Please take note of all the orgas, users and vms you've created -- this collection does not keep track of that** +--- + +This hacky solution will be replaced by a clean terraform abstraction ! + +--- ## Overview We use users, groups, resource pools, and roles to build a multi-tenante proxmox instance. -The idea is that virtual machines/containers are created by an admin and moved into a resource pool that is +Proxmox by design is not a "cloud native" hypervisor platform. Although it is possible to seperate users and resources from on another there is no way give users/organisations a quota of cpu,memory,disk from which they can assign virtual machines/containers to themselfs. + +To overcome this limitation an admin creates those vms/containers and then attaches them read-only to the specific user/organisation. + + +### Moving Parts + +For our hypervisor setup we need to administrate two components: + +* the hypervisor platform +* routing/switching + +Currently there is no idempotent automation for the hypervisor platform. State changes can be done via scripts or the cli directly (or via the WebGUI of course). + +The routing/switching side is fully automated. State changes are configured by modifying the `vlan_vms.yml` file. + +### Terminology + +Our abstraction is based on these objects: + +* users +* organisations +* vms + +Users can be part of multiple organisations. +Every vm is bound to a single organisation. + +### Proxmox Implementation + +In proxmox we use a few more components to implement our abstraction: + +* users +* groups (used as organisations) +* resource pools + * used to map permissions for groups +* acls + + +Under the (proxmox) hood our users (as in from our abstraction) are normal users. +Organisations are abstracted via groups. + +VMs and file-based storage are assigned to so called `Resource pools`. + +We now map permissions for a specific organisation (group) to a specific resource pool. This seperates the resources and creates a multi-tenant platform. + +There are WIP ## Requirements -* zfs storage pool for customer compartments: - * _change in `create_organisation`_ +HOW TO USE THESE SCRIPTS IN AN PVE ENVIRONMENT. +TBD -* storage pool for customer vm disks: - * `zfs create rpool/customer-disks` - * `pvesm add zfspool customer-disks --pool rpool/customer-disks --content images` - * _change in `create_vm`_ +## Create an Organisation -* bridge for customer vms - * _change in `create_vm`_ - * needs to be tag aware - * vms tag there traffic with the vmid +1. create proxmox related objects +``` +./create_organisation ${name} +``` -## Create a new VM -information you need to get beforehand from the custome: -- organisation -- username -- mail -**step 1: create an organisation** -* (if one does not already exist) -* `./create_organisation ` +## Create a VM -**step 2: create the user account** +**prerequisites**: +* there should be a organisation created for that vm +* you should have a list of `authorized_keys` for that organisation saved as a file on that node +0. search for next useable vlan/vm-id in `vlan_vms.yml` + * either use an old but freed id (block should be commented) + * or use next free one +1. check that we have the newest images +``` +grep debian download_images +ls /root/images +# execute ./download_images if script url does not match images directory +``` + +2. create vm +``` +./create_vm ${vmid} ${orga}-{vmname}-${vmid} ${orga} +./get_linklocal_for_vm ${vmid} +``` + +3. add vlan configuration to `vlan_vms.yml` and deploy ansible +``` +vim vlan_vms.yml +[...] copy block or create new one - change prefixed accordingly +[...] enter ll for v6 prefix +# deploy routers/switches +ansible-playbook -CD cores.yml -l cores,core_switches +ansible-playbook -D cores.yml -l cores,core_switches +# deploy hyperjump +ansible-playbook -CD vms.yml -l hyperjump +ansible-playbook -D vms.yml -l hyperjump +``` + +4. configure cloud-init settings for vm +``` +qm set ${vmid} --ipconfig0 "ip=${v4-cidr},gw=${v4-gw},ip6=${v6-prefix},gw6=fe80::1" +qm set ${vmid} --sshkeys ${path_to_authorized_keys} +``` + +5. change vm specifications to order +``` +# increase memory / cpu +qm set ${vmid} -memory 8192 --cores 4 + +# add more storage +# use customer-disks on hyper01 and hyper03 +# use customer-disks-slow on hyper02 (dedicated sata ssd pool with more space but less bandwith) +qm set ${vmid} --virtio1 customer-disks-slow:250,iothread=1,discard=on +``` + +6. start vm +``` +qm start ${vmid} +``` + +7. check reachability of vm +``` +ping6 ${v6prefix} +nc ${v6prefix} 22 +``` + +8. notify customer + +``` +vmid: ${vmid} +vmname: ${vmname} +user: debian +authentication: key based auth only + +v4: ${v4-cidr} +v4gw: ${v4-gw} +v6: ${v6-prefix} +v6gw: fe80::1 + +v6 reachable from everywhere - v4 only inside the reudnetz +ssh reverse proxy (for v4 access only) available via: ssh -p 2${vmid} @hyperjump.reudnetz.org + +best effort service - expect downtimes - backup your stuff ! +``` + +## Destroy a VM + +commands need to be executed on node where vm resides + +1. schedule removal of replicated storage (if configured) +``` +pvesr list | grep ${vmid} +pvesr delete ${vmid}-${replication_id} +echo wait till removal is complete - check with pvesr list +``` +2. remove vm +``` +qm shutdown ${vmid} +qm destroy --purge 1 ${vmid} +``` +3. remove vms from `vlan_vms.yml` and deploy +``` +vim vlan_vms.yml +[...] +# remove router and switch config +ansible-playbook -CD cores.yml -l cores,core_switches +ansible-playbook -D cores.yml -l cores,core_switches +# remove ssh reverse proxy config +ansible-playbook -CD vms.yml -l hyperjump +ansible-playbook -D vms.yml -l hyperjump +``` + +## Destroy User + +``` +./delete_user ${username}@pve +``` + +## Destroy organisation + +1. remove orga and associated objects from proxmox cluster +``` +./delete_organisation ${org} +``` +2. destroy remaining zfs volumes on other nodes +``` +# execute on all pve nodes +zfs destroy -r rpool/customer/${orga}-images +``` + +## Add/Remove user from organisation + +1. check which groups the user currently belongs to +``` +pveum group list | grep ${user}@pve +``` +2. change groups memberships +``` +# add to group +pveum user modify ${user}@pve --groups ${currentgroups},${new_group} +# remove from group +pveum user modify ${user}@pve --groups ${groups-without-other-group} +```