Welcome to Windmill's first Launch Week!
During the week of November 20, 2023, every day we will announce features that make Windmill even more reliable, faster, and easier to use.
A summary will be updated progressively on this page.
Day 1
Flow Builder Copilot
The very first feature of this launch week is Windmill Flow Copilot AI. The Copilot AI can generate the full flow including the code from a few prompts but also do it step-by-step.
It saves time for all users by returning a usable version of the flow in seconds, while leaving room for customization as each step is in the form of code. For less technical users, it's an opportunity to use Windmill while avoiding back-and-forth with Chat-GPT.
- Blog post
- Docs - Windmill AI for Scripts
- Docs - Windmill AI for Flows
- Docs - Windmill AI Code Completion
VS Code extension
Second annoucement of the day is the Windmill VS Code extension. It unlocks fully the ability to iterate on your scripts and flows in the comfort of your VS Code editor. It integrates particularly well with the current github/gitlab sync: scripts', flow steps' code and flows' yaml definitions can be directly edited in VS Code while benefitting from the instant preview UI of windmill.
- Blog post
- Docs - Windmill VS Code extension
- Docs - Local development
- Docs - Windmill CLI
Day 2
Worker groups management UI & Hot Reloading
Today's first annoucement of the day is the power of Worker Groups Configurations UI Management & Hot Reloading.
Workers are the foundation of Windmill. They are autonomous processes that run one script at a time using the entire cpu and memory available to them.
By default, every worker is the same and interchangeable. However, there are often needs to assign jobs to a specific worker pool, and to configure this worker pool to behave specifically or have different pre-installed binaries. To that end, we introduce the concept of "worker groups".
You can assign worker groups to flows and flow steps to be executed on specific queues, and for workers to only process jobs coming from specific queues.
Examples of worker groups configurations include:
- Assign different jobs to specific worker groups by giving them tags.
- Set an init script that will run at the start of the workers (e.g. to pre-install binaries).
- Dedicate your worker to a specific script or flow, for high throughput.
Once the configurations are done from the Windmill UI, the workers take their configuration dynamically, without having to restart.
Dedicated Workers for scripts
Dedicated Workers for scripts enable to reach up to 1000rps per worker for a given script. It makes Windmill faster than AWS Lambda and a good fit for even event-streaming use cases.
Day 3
Why is Windmill the fastest self-hostable workflow engine and job processing framework
Big claim today: We've benchmarked Windmill to be the fastest self-hostable generic workflow engine among Airflow, Prefect and even Temporal. For Airflow, there is a quite a margin, up to 10x faster!
You've known Windmill to be a productive environment to monitor, write and iterate on workflows, but we wanted to prove it's also the best system to deploy at scale in production.
It was important for us to be transparent and you can find the whole benchmark methodology here.
We also took the time to explain in details how we were able to achieve this.
Enjoy the reading.
- Blog post
- Docs - Benchmarks
Day 4
Day 4 of our Launch Week is dedicated to the App editor. The app editor is a low-code builder to create custom User Interfaces with a mix of drag-and-drop and code.
Global Custom CSS
Custom styling is essential for aligning internal tools with your brand image. While component-level CSS has always been possible in Windmill App editor, we are excited to introduce global CSS.
CSS can now be defined globally: for an entire app and across all apps as themes. This CSS is editable from a new CSS editor, a centralized location for defining, managing, and applying CSS rules.
With this feature, you can ensure consistent styling and create reusable themes for different teams, projects, clients etc. Not only that, but we also took care of doing an exhaustive hierarchy of classes that correspond to the strucuture ou our components, enabling to customize any aspect of them.
- Blog post
- Docs - Styling
- Docs - CSS editor
Groups of components
Windmill already provides over sixty components for its users. All components cover vast majority of needs. Components are most powerful when grouped together, enabling to implement complex logic.
We are proud to introduce component groups, which allow you to build your own meta-components, accessible via drag-and-drop from any app in your workspace. Saved components group keep the logic between each components, avoiding creating it over and over.
- Blog post
- Docs - Groups of components
Custom React Components & Apps
Windmill already provides over sixty components for its users. They should cover a vast majority of needs.
However, for components that are more specific you can now go even further and import your own components and apps written in React.
Once imported to Windmill, custom React components will be able to interact with the rest of the app.
There are 3 main benefits of using your own components:
- Reusing react components you've already written in other codebases.
- Controlling fully the component including theming, inputs and outputs allowing for more complex components and interactions.
- Embeding very complex components such as a code editor.
If that's not sufficient you can even build your own app in React.
- Blog post
- Docs - Import custom React components
- Docs - React app import
Day 5
For the last day of our launch week today we focused on features that will help you in your ETLs, with restartable flows and Workspace object storage for data pipelines.
Windmill for data pipelines - Workspace object storage
Run your ETLs on-prem up to 5x faster using Windmill compared to Spark while simplifying your infra.
An ETL is nothing else than a DAG of jobs, each of them reading datasets as input, running computation, and producing new datasets (or updating them).
Windmill enables building fast, powerful, reliable, and easy to build data pipelines:
- The DX in Windmill allows you to quickly assemble flows that can process data step by step in a visual and easy-to-manage way;
- You can control parallelism between individual steps, and set concurrency limits in case external resources need are fragile or rate limited;
- Windmill flows can be restarted from any step, making the iteration process of building a pipeline (or debugging one) smooth and efficient;
- Monitoring is made easy with error and recovery handlers.
For storage, you can now link a Windmill workspace to an S3 bucket and use it as source and/or target of your processing steps seamlessly, without any boilerplate.
The very large majority of ETLs can be processed step-wise on single nodes and Windmill provides (one of) the best models for orchestrating non-sharded compute. Using this model, your ETL will see a massive performance improvement, your infrastructure will be easier to manage and your pipeline will be easier to write, maintain, and monitor.
- Blog post
- Docs - Persistent Storage
Restartable Flows
The principle is simple: once the flow is executed (whether in test or deployed), you have the possibility to restart it from any node (step, iteration of a for loop, or branch).
For testing, it's great because you can restart from a specific node without recomputing the flow.
For prod, it's great because you can skip the already completed part of the flow avoiding duplicate actions (in particular, ensure data integrity).
This feature is part of the logic of the different ways to iterate over a flow (testing the entire flow, only one step, or up to a certain step).
Subscribe to follow our latest announcements
If you want to follow our announcements, sign up to Windmill cloud or to our mailing list below:
You can unsubscribe at any time.
You can self-host Windmill using a
docker compose up, or go with the cloud app.