SQLMesh is a next-generation data transformation framework designed to ship data quickly, efficiently, and without error. It enables data teams to run and deploy data transformations written in SQL or Python with visibility and control, supporting concepts like virtual data environments, automated testing, and CI/CD. It is backwards compatible with dbt and focuses on semantic understanding of SQL. The project maintains an active development cycle with frequent releases, with 0.234.0 being the latest stable version.
pip install sqlmeshVerified import paths — ran on the pinned version, not inferred.
The most common way to get started with SQLMesh is via its CLI. This quickstart demonstrates how to initialize a new project using DuckDB as the local engine and then run your first plan. The `sqlmesh init duckdb` command scaffolds a project, and `sqlmesh plan` shows the proposed changes. For a fully runnable example demonstrating project setup and execution, refer to the official documentation.
Always check the SQLMesh and SQLGlot changelogs before upgrading. Test your SQLMesh project thoroughly after an upgrade, especially if it involves `sqlglot` bumps, to catch any parsing or transpilation regressions.
Ensure your `pandas` version is compatible with your SQLMesh installation. Check the official SQLMesh documentation or GitHub issues for known `pandas` compatibility concerns and required version ranges.
Configure a persistent and reliable OLTP database (e.g., PostgreSQL) for SQLMesh state in production environments by specifying `state_connection` in `config.yaml`.
Always explicitly specify your development environment when running `sqlmesh plan` (e.g., `sqlmesh plan dev_myuser`). Consider changing the default environment in your project's `config.yaml` if all users work in isolated dev environments.
Use `sqlmesh create_external_models` or manually define external models (e.g., in `schema.yml`) to provide SQLMesh with schema information for external data sources. This enables full column-level lineage and better optimization.
Explicitly define column names and types in your SQL models, particularly in the final `SELECT` statement. Avoid `SELECT *` in production models where schema stability is critical. Manually specify `columns` in the model definition if schema inference is problematic.
pip install sqlmesh
Verify the PostgreSQL server is running and accessible, and check the connection details (host, port, user, password) in your `sqlmesh.yaml` configuration.
Ensure the referenced model (`my_model_name`) is correctly defined, its file is in the project path, and there are no typos in its name.
Carefully review the Jinja template syntax in your SQLMesh model, paying attention to blocks, variables, filters, and macro calls.