🎥 Recordings from CozySummit Virtual 2026

Treating Kubernetes as a Linux Distro: APT-Style Packaging with FluxCD | Andrei Kvapil, CEO, AENIX

00:00 - What Cozystack is and its managed services focus
01:14 - What makes a service truly managed
02:23 - Platform stack foundations: storage, networking, virtualization, and monitoring
03:34 - Cozystack as a tested Kubernetes distribution
04:16 - Monorepo structure and package organization
05:06 - System, apps, and extra charts
06:28 - Helm charts as packages
07:08 - Delivering Cozystack through Flux and OCI artifacts
08:42 - Platform chart and configuration storage
09:25 - Previous bundle-based delivery model
10:18 - Customizing charts and delivering ad hoc fixes
10:52 - Local development workflow challenges
11:32 - Cozy chart tool and faster local testing
12:33 - Show, diff, and apply commands
13:36 - Simple installation and air-gapped delivery goals
14:20 - Cilium, kube-proxy, and the chicken-and-egg problem
15:06 - Flux all-in-one and early bootstrap networking challenges
16:54 - Moving from in-cluster HTTP server to OCI artifacts
17:26 - Limitations of OCI-backed multi-chart Helm repositories
18:50 - Source Watcher and artifact composition
20:04 - Flux controller responsibilities and artifact generators
20:53 - Combining sources into external artifacts
21:45 - Making Cozystack reusable for different customers
22:23 - Package manager idea inspired by Alpine Linux
22:55 - Package sources, packages, and Helm releases
24:04 - Package variants and multi-chart packages
25:06 - Package dependencies and dependency checks
26:30 - Cozy package CLI commands
27:04 - Migration from bundles to packages
28:04 - Platform package variants like Ubuntu Desktop and net install
28:33 - Scaling Cozystack without feature bloat
29:41 - External repositories and vendor-distributed packages
30:35 - Evolution toward an ecosystem model
32:22 - Kubernetes as the new Linux and ecosystem-driven distributions
32:43 - Helpful links and documentation
33:32 - Q&A begins
33:39 - State management and disaster recovery over time
35:07 - MetalLB vs Cilium L2 announcements
36:26 - Secure Boot support
37:18 - Cozystack software development lifecycle
38:25 - Learning resources for Kubernetes and Cozystack

Enjoy watching and share your thoughts: https://www.youtube.com/watch?v=vKRZR7gjU40

🧿 How LINSTOR and DRBD Keep Your Data Alive When a Server Dies

Every bare-metal operator's nightmare: a disk fails at 3 AM. If your storage is node-local (a local-path or hostPath volume pinned to one node), you're restoring from backup and praying it's recent. Ceph gives you replication but demands 3+ dedicated storage nodes, a PhD in CRUSH maps, and constant tuning. Most teams either over-invest in storage infrastructure or under-invest and pay the price in outages.

Cozystack uses LINSTOR + DRBD for replicated block storage. DRBD replicates at the block device level — below the filesystem, below the database, below everything. It's been battle-tested in Linux HA setups for over two decades. LINSTOR adds the Kubernetes-native orchestration layer on top.

⚙ How It Works

1. DRBD (Distributed Replicated Block Device) synchronously mirrors block devices between nodes. Every write to Node A is immediately written to Node B before the application gets an acknowledgment.
2. LINSTOR manages DRBD resources as Kubernetes StorageClasses. When a PVC requests storage, LINSTOR creates a DRBD volume, selects replica nodes, and handles the lifecycle.
3. CSI Driver exposes LINSTOR volumes as standard Kubernetes PersistentVolumes.

Result: any pod or VM that uses a replicated StorageClass gets synchronous block-level replication across nodes — transparently.

🎛 Setting Up Storage

Step 1 — Prepare disks:

# Set up LINSTOR CLI alias
alias linstor='kubectl exec -n cozy-linstor deploy/linstor-controller -- linstor'

# Check what LINSTOR sees
linstor node list
linstor physical-storage list

If disks don't appear, they likely have leftover metadata. Clean them:

# Using talm (installation covered in POST 9)
# WARNING: never wipe the OS install disk (machine.install.disk) — only data disks.
talm -f nodes/node1.yaml wipe disk nvme0n1 nvme1n1

Step 2 — Create storage pools (ZFS):

linstor physical-storage create-device-pool \
  zfs node1 \
  /dev/nvme0n1 /dev/nvme1n1 \
  --pool-name data \
  --storage-pool data

Repeat for each node.

Step 3 — Verify:

linstor storage-pool list

You should see a data pool on each node with available capacity.

Step 4 — Use it:

Any application that specifies storageClass: replicated now gets DRBD-replicated volumes (three replicas by default). PostgreSQL, MongoDB, VM disks — all of them.

# In any application config:
values:
  storageClass: replicated
  size: 50Gi

🛟 What happens when a node fails:

1. DRBD detects the node is gone
2. The volume remains available on the surviving replica node
3. Kubernetes reschedules the pod to a node that has the replica
4. When the failed node comes back, DRBD automatically resyncs

No manual intervention. No restore from backup. No data loss.

📘 Documentation

- Disk Preparation: https://cozystack.io/docs/storage/disk-preparation/
- Disk Encryption: https://cozystack.io/docs/storage/disk-encryption/
- DRBD Tuning: https://cozystack.io/docs/storage/drbd-tuning/
- NFS (RWX): https://cozystack.io/docs/storage/nfs/

Join the community:
Telegram group t.iss.one/cozystack
Slack group https://kubernetes.slack.com/archives/C06L3CPRVN1
(Get invite at https://slack.kubernetes.io)

Cozystack resources:
https://cozystack.io
https://cozystack.io/docs/get-started
https://cozystack.io/blog
https://github.com/cozystack/cozystack

#Cozystack #LINSTOR #DRBD #Kubernetes #CloudNative #BareMetal #PrivateCloud #Storage

Hi everyone, if you'd like to add anything to tomorrow's Cozystack community meeting agenda, feel free to reach out to @vsop78 or simply drop your ideas in the comments

When: Tomorrow at 18:00-19:00 CET
Where: Zoom
Link to join in:
https://zoom-lfx.platform.linuxfoundation.org/meeting/93845795591?password=a263fc60-ea72-41c8-84c8-a00d683ecee5

This meeting is free and open to everyone!
Come and join us!

Catch you at the Cozystack community meeting today!

When: Thursday at 18:00-19:00 CET
Where: Zoom
Link to join in:
https://zoom-lfx.platform.linuxfoundation.org/meeting/93845795591?password=a263fc60-ea72-41c8-84c8-a00d683ecee5

This meeting is free and open to everyone!
Come and join us!

Minutes:
https://docs.google.com/document/d/1YObSKsQ3Gzi1gSLPlUp-PbSYEeVkoyzY20M_hXKVhq4/edit?tab=t.0#heading=h.ygb3chatmgn2

📢 Agenda and Notes:
- Cozystack v1.6.0 release
- Public IPs as First-Class Resources in Cozystack by Timofei Larkin (lllamnyp) https://github.com/cozystack/community/pull/35

🎤 Open Floor:
- How to make Kubernetes cheaper on resources: where node overhead actually lives
by Malformed C, maintainer of Periapsis
- Test flakiness and platform stability: catching component conflicts earlier, isolating tests from external factors
- Discussion of versioning and release process
- Release frequency, semver vs calver and
- Backport strategy discussion
- Design proposal review: open proposals on GitHub
- Viewing backup process events in the web dashboard
- Marketplace service — discussion


🗓️Do not miss Cozystack community meetings:

https://webcal.prod.itx.linuxfoundation.org/lfx/lfsixxnFWxbvsyEuC2

https://zoom-lfx.platform.linuxfoundation.org/meetings/cozystack?view=month

💡Have a feature idea or design proposal for Cozystack?
Submit your Pull Request here: https://github.com/cozystack/community/pulls 🚀

🎥 Cozystack community meeting 2026-23-07

https://youtu.be/vA8ilRMQz0k

Cozystack: Free Cloud Platform based on Kubernetes | CNCF Sandbox Project

Minutes: https://docs.google.com/document/d/1YObSKsQ3Gzi1gSLPlUp-PbSYEeVkoyzY20M_hXKVhq4/

📢 Agenda and Notes:

- Cozystack v1.6.o release
- Cozystack preparation for CNCF Incubation
- Public IPs as First-Class Resources in Cozystack by Timofei Larkin (lllamnyp) https://github.com/cozystack/community/pull/35


🎤 Open Floor:

- How to make Kubernetes cheaper on resources: where node overhead actually lives by Malformed C, maintainer of Periapsis
- Marketplace service — discussion


🖖 Share your enhancement and architecture proposals by opening a PR at https://github.com/cozystack/community/pulls

🫵 Join the community:
Telegram group t.iss.one/cozystack
Slack group https://kubernetes.slack.com/archives/C06L3CPRVN1
(Get invite at https://slack.kubernetes.io)

Cozystack resources:
https://cozystack.io
https://cozystack.io/docs/get-started
https://cozystack.io/blog
https://github.com/cozystack/cozystack

📅 Subscribe to our community meetings calendar: https://zoom-lfx.platform.linuxfoundation.org/meetings/cozystack

🗓️ Add meetings to your calendar: https://webcal.prod.itx.linuxfoundation.org/lfx/lfsixxnFWxbvsyEuC2

https://youtu.be/vA8ilRMQz0k

Focus on today: how Andrei Kvapil @kvaps, the founder of Ænix and the creator of Cozystack, an open source cloud platform, built aeman, a daily planning board for engineers on top of GitHub Projects.

Read the full article in Ænix blog 👇👇👇

https://blog.aenix.io/focus-on-today-how-we-built-aeman-a-daily-planning-board-for-engineers-on-top-of-github-projects-c59da4451b8b

🎥 Recordings from CozySummit Virtual 2026

What If Every Cozystack Change Became a Commit? | Simon Koudijs, configbutler.ai

00:00 - Speaker intro and Config Butler background
00:47 - First impressions of Cozy Stack
01:19 - Resource history problem: who, when, and why
01:50 - Options for handling unmanaged platform changes
02:18 - Kubernetes audit files and audit webhooks
02:48 - Classic GitOps and its interface tradeoffs
03:41 - Live demo setup on a local Cozy Stack cluster
04:25 - Audience quiz submission demo
05:37 - Quiz submissions as Kubernetes custom resources
06:24 - Capturing platform changes as YAML in Git
06:41 - Classic GitOps vs reverse GitOps
07:13 - Reverse GitOps principles
08:05 - GitOps Reverser implementation overview
08:56 - Watch rules, Git targets, and Git providers
10:03 - Live Git commits from audience submissions
10:55 - Git author vs committer attribution
11:16 - Capturing the reason behind changes
11:29 - Coffee ordering demo intro
12:01 - Application artifacts and configuration state
12:32 - Coffee ordering and admin configuration demo
13:27 - Handling concurrent edits
14:22 - Completing the who, when, and why audit trail
14:50 - Why Cozy Stack fits reverse GitOps well
15:32 - Cozy Stack dashboard identity forwarding demo
16:21 - Capturing high-level resources and Flux changes
17:14 - Watch vs admission webhooks vs audit webhooks
17:43 - Benefits: auditability, GitOps on-ramp, and GUI usage
18:37 - Setup complexity and Kubernetes API limitations
19:10 - Secrets encryption and avoiding inline passwords
19:51 - Feedback request and future ideas
20:27 - Key takeaway: Kubernetes API as record of intent
20:54 - Links and feedback form
21:39 - Q&A begins
21:46 - Why not just snapshot or back up ETCD
22:22 - Server-side apply and field owners question
22:39 - YAML audit trail and resource history
22:52 - Git history length and conflict handling
24:00 - Conflict resolution strategy
24:31 - API interface for viewing Git changes
24:48 - Handling Git outages with Redis cache

Enjoy watching and share your thoughts: https://youtu.be/dkyjOq-I7Zs?si=qIwL5IiIU8ZxPArq

New on the Kubernetes blog: Andrei Kvapil /@kvaps and Timofei Larkin/@lllamnyp (Ænix) break down how the controller-runtime cache actually works — and why your controller isn't crashing the API server even at hundreds of reconciles per second.
If you write controllers in Go, this builds the mental model most people are missing: reads come from a local cache, writes go to the API server, and almost every production surprise (memory blowups, stale reads, silent O(n) scans) follows from that one fact.
https://kubernetes.io/blog/2026/07/29/controller-runtime-cache-explained/

🚀Cozystack v1.6.0 is now available. The release was published on July 22, 2026, and includes all fixes previously shipped in the v1.5.1, v1.5.2, and v1.5.3 patch releases.

⚙️This release changes several important parts of the platform. Tenant Kubernetes workers now run Talos Linux instead of Ubuntu, tenants can enable OIDC authentication for Kubernetes and Grafana, and a new SecurityGroup API provides a safer interface for managing application network policies.

🧩Cozystack 1.6 also introduces hierarchical resource quotas, completes the etcd-operator v1alpha2 migration, expands Keycloak security and backup options, and makes application deletion consistently reclaim its storage.

👀The upgrade surface is larger than usual.

🧿Operators should review the upgrade section before applying the release.

Read the full release information in our social media👇
https://x.com/aenix_io/status/2082778204167868599 or
https://www.linkedin.com/pulse/cozystack-16-talos-workers-tenant-sso-security-groups-hierarchical-jkznf

😎Your infrastructure is already running.

💰But can your customers actually buy cloud services from it?

On August 19, Andrei Kvapil, creator of Cozystack, founder of AEnix, will walk through the full path from servers to a sellable public cloud:

• VMs, managed Kubernetes, databases and S3 storage
• Billing, usage metering and hosting panels
• WHMCS and custom integrations
• Migration from VMware, OpenStack or a custom stack
• A realistic two-week path to your first customers

Free live webinar.

Questions submitted during registration get priority in the live Q&A.

August 19 · 10:00 CET · Online

Register even if you cannot attend live. We will send you the recording.
Registration is now open!👇
https://aenix.io/webinars/launch-public-cloud/

New Cozystack Highlights: Persistent Public IPs.

Today IP addresses in Kubernetes are ephemeral. You create a LoadBalancer service, you get an IP, you delete the service, the IP is gone. Recreate it and you get a different one. For anyone running real workloads on public IPs, that churn is a problem, and Cozystack users have been asking to fix it for a while.

The prototype shown at the last community meeting borrows the model everyone already knows from storage: just like a PersistentVolumeClaim binds to a PersistentVolume through a StorageClass, an IPAddressClaim binds to an IPAddress through an IPAddressClass.

What that unlocks:
— Reserve an IP before attaching it to anything, then bind it to a service via annotation
— Delete the service, the IP stays reserved and comes back to the same claim
— Move one reserved IP between services without losing it
— Contention policies: keep an IP alive after its claim is deleted, or release it back to the pool
— A reserved IP can't be stolen by another service

The design is deliberately platform-agnostic. The core controller handles the claim-to-address logic; a separate driver talks to the backend, MetalLB today, but the same pattern fits AWS Elastic IPs or anything else, much like a CSI driver for storage.

Still a prototype, but the direction is set: IP addresses become a first-class, reservable resource in Cozystack.

🎥 Full demo from the community meeting: https://www.youtube.com/watch?v=cQJyA25Hf6U

#Cozystack #Kubernetes #Networking #CloudNative #OpenSource

Hi everyone, if you'd like to add anything to tomorrow's Cozystack community meeting agenda, feel free to reach out to @vsop78 or simply drop your ideas in the comments

When: Tomorrow at 18:00-19:00
(Central European Time, CET)
Where: Zoom
Link to join in:
https://zoom-lfx.platform.linuxfoundation.org/meeting/93845795591?password=a263fc60-ea72-41c8-84c8-a00d683ecee5

This meeting is free and open to everyone!
Come and join us!

Catch you at the 70th Cozystack community meeting today!

When: Thursday at 18:00-19:00 (Central European Time, CET)
Where: Zoom
Link to join in:
https://zoom-lfx.platform.linuxfoundation.org/meeting/93845795591?password=a263fc60-ea72-41c8-84c8-a00d683ecee5

This meeting is free and open to everyone!
Come and join us!

Minutes:
https://docs.google.com/document/d/1YObSKsQ3Gzi1gSLPlUp-PbSYEeVkoyzY20M_hXKVhq4/edit?tab=t.0#heading=h.ygb3chatmgn2

📢 Agenda and Notes:
- Marketplace service in details

🎤 Open Floor:
-Test flakiness and platform stability: catching component conflicts earlier, isolating tests from external factors
- Discussion of versioning and release process
- Release frequency, semver vs calver and
- Backport strategy discussion
- Design proposal review: open proposals on GitHub https://github.com/cozystack/community/pulls
- Viewing backup process events in the web dashboard


🗓️Do not miss Cozystack community meetings:

https://webcal.prod.itx.linuxfoundation.org/lfx/lfsixxnFWxbvsyEuC2

https://zoom-lfx.platform.linuxfoundation.org/meetings/cozystack?view=month

💡Have a feature idea or design proposal for Cozystack?
Submit your Pull Request here: https://github.com/cozystack/community/pulls 🚀

📣 This August, the Ænix team is bringing a series of hands-on open source private cloud workshops to four cities across Central Asia: Almaty, Bishkek, Tashkent and Astana.

The workshop focuses on moving beyond traditional virtualization and exploring cloud-native infrastructure for teams evaluating alternatives to VMware.

❓ We’ll cover:

• how a modern cloud-native private cloud is built;
• how VMware components map to an open source stack;
• where Kubernetes, KubeVirt, LINSTOR, Cilium and GitOps fit in;
• how to migrate virtual machines and introduce self-service infrastructure;
• when migration makes sense and when keeping your existing VMware environment is the better choice.

Cozystack will be used as a hands-on example of an integrated platform for testing these concepts and migration scenarios in practice.

🗣️ The workshops will be led by Timur Tukaev, COO at Ænix, the team behind Cozystack.

📍 Almaty — August 13, 10:00–14:00
📍 Bishkek — August 17, 14:00–18:00
📍 Tashkent — August 20, 15:00–19:00
📍 Astana — August 24, 14:00–18:30

All workshops are in person and follow the same program.

🔗 Choose your city and register via the link: https://u.to/VpqnIg

🙌 Know someone in Almaty, Bishkek, Tashkent or Astana who might be interested?
If you have partners, colleagues or friends working with cloud infrastructure, virtualization or Kubernetes in these cities, please share this post with them.

We’ve open-sourced another tool we built: keycloak-kms-proxy.
Here’s the problem it solves.

Keycloak stores user PII, including email addresses, names and custom attributes, as plaintext in its database. Under GDPR, this is personal data and needs proper protection at rest.

Storage-level encryption or PostgreSQL-side encryption only gets you so far. The encryption key still lives close to the data, often inside the same PostgreSQL environment. Key rotation is also difficult to handle without downtime.

There are unofficial plugins that address this, such as Keycloak PII Data Encryption Provider. But they come with their own trade-offs: tight coupling to specific Keycloak versions, Java-level integration and a static encryption key passed through an environment variable.

We took a different approach. keycloak-kms-proxy is a transparent proxy that operates at the PostgreSQL wire-protocol level. It sits between Keycloak and PostgreSQL. Keycloak sees a regular Postgres database, while the proxy encrypts selected columns on write and decrypts them on read.

The design follows the same basic principles as Kubernetes KMSv2.
Every encrypted value is self-describing ($KKP$...), which means the proxy can safely work with a partially encrypted database while the backfill utility gradually processes existing records.

KEK rotation in Vault is fully transparent. The key version is stored alongside each wrapped DEK, so existing data remains decryptable without migrations or downtime.

In Cozystack, this is already available as an optional feature in the system Keycloak package. Enable encryption in the Helm chart and the proxy is deployed automatically. Keycloak is then reconfigured to connect through it, with no additional setup required.