On 20.6.2024, five years after writing the first line of code and a year and a half after the project was shut down by the corporate owner, we re-launched Zerops as a standalone startup company, headed by the original dev team.

Since then we've been hard at work adding support for new services, improving developer experience, flexibility as well as our infrastructure layer. Our Discord grew to over 500 developers and over 2000 developers gave Zerops a try. We got some great, encouraging feedback and our users are deploying everything from simple static websites to outrageously complex large projects with multiple databases and tens of microservices, confirming we've hit the sweet spot between flexibility, scalability, developer experience and affordability we were aiming for. To share two pieces of feedback with you:

I think the price/quality ratio is really good. Before we were playing with Kuberenetes cluster on some 10+ VPSs. Even though for the same price we get more resources, it takes so much time to setup and maintain, plus we'd probably need a separate person to maintain it in the future. Even though we could get it on AWS, Google or Azure, it's so much more expensive and complicated. Your solution hits the mark with ease of use and price.
Karolis

I'm currently using Caprover plus some self managed VMs on Hetzner. Over the last 10 years this project went from Heroku -> AWS -> Scalingo -> Clever Cloud -> DigitalOean -> Caprover+Hetzner. I literally stumbled over Zerops through the Gleam people and I think it hits the sweet spot between a Caprover/Hetzner and an overpriced PaaS... so far, needless to say, I'm pretty impressed by what you guys have built.
Louis

Let's recap everything we've added and improved over the past 6 months and take a high level look at our plans for 2025.

Table of contents

• Free Lightweight project core
• Backups
• Virtual machines + Docker
• Remote web shell + starting empty containers without user's code
• Access to build container + pipeline debug options
• Service for static apps
• Project environment variables
• Multiple start commands
• CRON
• Speed improvements
• 14 new services, 14 new recipes
• CLI, GUI, Docs improvements
• Infrastructure improvements
• Analog.js and spartan.ng partnership

Free Lightweight project core

When a project was created in Zerops, a dedicated "core" infrastructure was deployed along with it, in practice this meant:

• two containers (active/active backup) with L3 balancer (has unique IP addresses assigned, accepts traffic from the Internet and forwards it to the services inside)
• one container that gathers resource statistics from all services
• one container that gathers and optionally forwards logs from all services inside the project.
• then there was highly available (active/active) L7 balancer that received http/https traffic from the L3 balancer, handled SSL termination and issuing of SSL certificates

These containers had a total of 5 CPU cores and 2.5 GB of RAM, but would also automatically scale as traffic to your own services inside the project increased. This cost a flat fee of $5/30 days. But not just for resources, that fee also included fair use based build time on 4 CPU / 8 GB RAM build containers, egress and free proxy/load balancers for HA databases services.

All in all, it was a lot of bang for the buck, we were giving the project core at cost price, because we wanted really high grade dedicated infrastructure setup anyone could use even if it was for a small application. But there was one problem. If you deployed a single small static / fullstack site, you'd pay $1.5/month for the service + $5/month for the core, making it costly and inefficient for this use case for both Zerops and the user. Similarly, if you ran a development environment of even a bigger project, you could easily run all your services on the lowest resources possible, but you were still paying for high-grade production project core you didn't really need.

Given this, we've decided to introduce two project core packages:

Free "lightweight" project package

This has the same exact capabilities and technologies as the legacy core, but instead of running on 5 different containers, it runs on a single container, making it a great fit for smaller projects and non-production environments.

$10/month "serious" project package

For production environments of larger projects, or a project that includes many small apps

Along with the difference in the way core services are created, the serious package also includes a generous amount of build time and egress, while the free / lightweight package includes a limited amount. Long term, the excess in build time / egress will be billed on hourly (build) / GB (egress) basis, with prices lower than what competitors offer. For now we'll just monitor the usage without billing the excess.

Backups

Zerops previously utilized snapshot backups of our whole physical nodes, this made restoring data cumbersome for both our support and our users. So at last we implemented a native backup mechanism and users now can define backups per service, set their frequency and trigger the backup manually. Backups are then encrypted and stored in a separate backup cluster.

Virtual machines + Docker

When we started developing Zerops back in 2019, we made the conscious decision to build it on top of LXD (now Incus), instead of building on top of Kubernetes or service offerings from one of the major cloud providers (AWS, GCP, Azure). There are multiple ways to handle virtualization and containerization, to give you a high-level overview:

• Full VMs: Complete freedom to run any workload with perfect isolation, but the operational overhead makes iterative development and scaling unnecessarily complex.
• MicroVMs: Perfect for compute-intensive, stateless workloads, but the optimization for fast startup makes them unsuitable for persistent services.
• System Containers: Excellent balance for running both stateful and stateless services with proper isolation, though requires more sophisticated management than pure application containers.
• Application Containers: Optimized for stateless application runtimes and microservices, but struggles with stateful services requiring deep system access or consistent performance.

For what we wanted to achieve – a platform with enterprise-grade infrastructure setup, horizontal and vertical autoscaling, offering not only runtimes but highly available databases, stores, and storage as well – system containers were the obvious choice. And so we built on top of Incus, created our own orchestration, abstracted a lot of the complexity away, and ended up with all the flexibility of system containers while exposing developer-oriented APIs. Amazing!

..but this had one major disadvantage: a lot of open source software simply gave up on providing a way to run on pure OS and opted for offering Docker images as the only way to run. While it's possible to run Docker on system containers, it's only possible in privileged mode which we can not use due to security reasons.

Luckily, over the years Incus evolved and now is able to spin up not only containers, but offers a way to run efficient, QEMU-based full VMs fully integrated into the Incus ecosystem. This means we were able to prepare a "virtual machine" type of service that can run inside the same private network as the other services, thus utilizing all the managed services, as well as offering nearly identical management/observability experience as our container-based offering, including remote web terminal, env variables management, logs, granular scaling configuration (except autoscaling - unlike containers, VMs cannot properly scale on the fly).

This VM is now utilized by our "Docker" service, and you get a running full VM with Docker ready to use in around 30 seconds.

We plan to use this service to provide one-click deployment of many popular open source software, but for user's applications, we still strongly suggest to use the native container-based Zerops services offering.

Remote web shell + starting empty containers without user's code

Zerops always had the option to securely ssh into any runtime container, after connecting to the project’s private network using the WireGuard VPN built into our CLI. But sometimes you just need to quickly spin up a container to test something and then the process becomes annoying, especially since you needed to trigger a pipeline with your code to actually start the container.

So we've added the option to start containers immediately after the service is added, without need to deploy your code first, and along with it, we've added a remote web shell built into our GUI, giving you a quick access into runtime containers without any additional requirements.

Access to build container + pipeline debug options

When you are deploying applications, the most annoying thing (not exclusive to Zerops) is when some build process works fine locally but doesn't in your CI/CD setup. You end up endlessly deploying new versions, ls-ing, pwd-ing, apt-ing, printing, and chmod-ing your way through with tens of pipeline triggers. It's frustrating no matter if you're trying out Zerops for the first time or are a power user (and believe me, we would know, being the biggest power users of our platform ourselves — the countless hours this has consumed over the years).

And so, building on top of the remote web terminal access to runtime containers we've added recently, we are introducing remote web terminal access to build & runtime prepare containers. To be able to properly debug, we've added settings for pipeline debug options, which allows you to control pipeline behavior, telling the process to stop at certain points (before, after commands, and on commands fail), giving you time to access the container, debug, run commands manually, and eventually even resume the process.

Native service for static apps

From the beginning Zerops offered an Nginx service to host your static apps, but Nginx configs are usually scary for frontend developers, so we’ve added a service that simplifies the configuration and offer similar configuration options as for example Netlify's netlify.toml.

Now setup for your standard Angular / react / vue etc. app is simple as using base: static. Redirect of non-existant files to index.html is included by default. This service can be exported to Nginx config any time.

zerops:
  build:
    ... 
  run:
    base: static

Project environment variables

Sometimes a variable is needed across different services inside the project, so we've added the ability to define environment variable on project level.

Multiple start commands

Zerops containers are based on Incus containers, placing them at a midpoint between containerized processes (e.g., Docker) and full-fledged VMs (e.g., Proxmox). This architecture allows Zerops to easily run multiple processes within a single container.

So we've added a new property to easily make use of this. As an alternative to defining run.start and run.initCommands, you can now use run.startCommands. Here's an example where it might be useful, starting a Node.js application that uses SQLite, along with Litestream to persist the data across deployments.

zerops:
  build:
    ... 
  run:
    ...
    # here instead of standard `start` property, we use `startCommands`
    # which allows us to define multiple commands (both start and init)
    # as well as name the set, to allow for convinient filtering in runtime logs
    startCommands:
      # start the application
      - command: npm run start:prod
        name: server

      # start the replication
      - command: litestream replicate -config=litestream.yml
        name: replication
        # restore the database on container init
        initCommands:
          - litestream restore -if-replica-exists -if-db-not-exists -config=litestream.yml $DB_NAME

Cron

Running Cron jobs on highly available containers can be tricky, because it would run on all containers and possibly cause unwanted behaviours, so we’ve added the ability to define Cron job in zerops.yml with crontab syntax and Zerops will make sure they'll execute only on one container.

zerops:
  build:
    ... 
  run: 
    ...
    crontab:
      - command: "php artisan schedule:run"
        timing: "* * * * *"
        workingDir: /var/www/html

Speed improvements

We started working on major improvements to the platforms speed, achieving up to 10x speed improvements in various parts of the platform - https://zerops.io/article/tracing-is-king-how-we-sped-up-our-paas-by-10x

14 new services, 14 new recipes

We’ve been steadily increasing the number of services Zerops supports, from runtime world we've added native support for Java, Bun, Deno, Exilir, Gleam, from managed services we’ve added support for the most popular search, messaging, KV stores and vector DBs — Elasticsearch, Valkey, Meilisearch, Typesense, NATS, Kafka, Qdrant. All available in high availability, with backups, failovers, healthchecks, repairs, scaling, configration and optimizations.

We've also added multiple new one-click recipes that showcase production grade integration of all the supported runtimes as well as representatives of the most popular frameworks for each runtime.

CLI, GUI, Docs improvements

We've also been hard at work improving UX and DX of both our CLI and the web application with tens of improvements of various size and impact.

Infrastructure improvements

We've worked on deep restructuring of our infrastructure layer. We standardized our historically complex systems, rebuilt our Ansible configuration with proper role-based architecture, and migrated everything to Noble. On the storage front, we replaced our legacy CEPH deployment with multiple MinIO clusters for better reliability and simpler maintenance. We've also overhauled our internal networking topology and certificate authority infrastructure.

On reliability front we implemented dual VPN infrastructure and rolled out a centralized authentication system for infrastructure services. For observability, we deployed Prometheus and Loki for metrics and logging visualization, and implemented distributed tracing with APM to gain deep visibility into service performance and bottlenecks. We also started work on our geo steered nginx caching layer - complete with a custom management API backed by Redis - to optimize content delivery. These changes weren't just about modernization - they directly addressed performance bottlenecks and operational complexities we'd encountered while scaling and managing the platform.

Analog.js and spartan.ng partnership

We became partners with Analog.js and spartan.ng with the goal of putting Angular's fullstack and components system against the community standard Next.js / shadcn combo. Demonstrated on real life use cases of the most popular open source software. Starting with the amazing Medusa.js soon.

What’s next?

Here's a high level roadmap of what we have planned for 2025

• CDN for static apps and object storage files
• service for VMs with GPUs
• migration containers
• pulumi support
• http routing improvements (caching support for wildcards)
• OpenTelemetry (statistics forwarding)
• auto start/stop
• customizable database services (installing plugins, changing core configurations)
• external access to database services
• new auth system with granular permissions
• new notification system with ability subscribe / unsubscribe to projects and multiple channels (slack, discord, mail, webhook)
• multiregionality