We are currently working on our managed Postgres offering. Should be available in technical preview in September. Other services like providing object storage are planned too but I don't have any ETA to share for now :)
Glad to hear that! We plan to provide object storage in the future. For now, our recommendation is to rely on a specialized provider for that, i.e. S3, GCS, R2, Wasabi, etc.
I would need a very good bandwidth between the instances and the object storage. On AWS, we manage to get 1 GB/s for one instance.
Is there one particular service that will perform particularly well? Also, I don't really want to pay for data transfer...
Minio is an awesome object storage solution with great features such as the S3 gateway but from what I know they don’t have a processing layer.
At Koyeb we also have an S3 compatible layer to let users send, process, and store (on any cloud or your own edge Minio) data. As @bdcravens said, its where similarities end, plus Koyeb is entirely managed.
We’re currently working on a deep Git integration where you basically push your updated workflow configuration with the code of functions/docker tag reference.
We have some tooling to develop individual catalog integrations locally and test the integration with object storage works as expected. We plan to publicly release this tooling for all users to be able to test their functions locally before using them on a workflow.
For workflows environment, currently, right now, you have to create one processing Stack for each environment, i.e. dev, staging, and prod. Later on, we want to spawn environements for each Git branch.
The `after` attribute is present to let you implement your processing logic. The steps list contains the processing actions but it is not a sequential execution. The workflow can have multiple processing branch and perform a series of processing on the result of a specific step.
Scaleway.com also offers baremetal servers at a really attractive price. The CLI is just awesome, it's great to see other cloud providers joining the game.
You end up having to build your own moving parts...
Building a high-availability metadata store is not easy. And ensuring that incoming request IPs aren't spoof is a little non-trivial to reason about.
UserData is a good way to provide a one-time token that can be used to fetch data...
Using SSH for provisioning is just plain dirty... and almost impossible to do reliably... You'll need global locks and timeouts to recover in case one of your master crashes... Plus some garbage collection to cleanup things that where not fully provisioned.
This is a LOT of unreliable state to manage. And ton of corner cases. Having the right architecture matters for reliable automation.