Dagster-Airbyte Integration Broken for OSS Version Due to Incorrect API Path (/api/v1 vs /api/public/v1) #32549
Description
What's the issue?
The current dagster-airbyte integration is incompatible with the latest open-source (OSS) version of Airbyte. The root cause is an incorrect API endpoint path.
The dagster-airbyte library and resources (specifically the AirbyteResource) are hardcoded to make requests to the Airbyte API at the base path /api/v1/. However, starting with a recent version of Airbyte OSS (the non-Enterprise edition), the API base path has been changed to /api/public/v1/.
This discrepancy results in all communication between Dagster and Airbyte failing with HTTP 404 Not Found errors, as the /api/v1/ endpoints no longer exist in the OSS version
What did you expect to happen?
The resource should connect successfully and be able to interact with the Airbyte API.
How to reproduce?
- Set up a recent OSS version of Airbyte (e.g., via Docker Compose).
- Configure an AirbyteResource in Dagster pointing to your Airbyte instance's base URL.
- Attempt to initialize the resource or materialize an asset that uses it.
- Observed Result: The operation fails with a connection error or a 404 for endpoints like /api/v1/health or /api/v1/workspaces/list.
- Expected Result: The resource should connect successfully and be able to interact with the Airbyte API.
Dagster version
latest
Deployment type
Local
Deployment details
No response
Additional information
No response
Message from the maintainers
- When will this be fixed? Is there an estimated timeline for a patch release that addresses this critical incompatibility?
- How can we fix it? If this is a known issue and a fix is in progress, is there a public branch or pull request we can follow?
- Will this be supported? Can you confirm that the dagster-airbyte integration will continue to be maintained for the open-source version of Airbyte?
- Is there a stable version? In the meantime, is there a known stable version combination of Dagster and Airbyte OSS that still works? For example, is using an older version of Airbyte that still exposes the /api/v1 endpoint a viable temporary workaround?