Deprecated: Function get_magic_quotes_gpc() is deprecated in /hermes/walnacweb04/walnacweb04ab/b2791/pow.jasaeld/htdocs/De1337/nothing/index.php on line 99

Deprecated: The each() function is deprecated. This message will be suppressed on further calls in /hermes/walnacweb04/walnacweb04ab/b2791/pow.jasaeld/htdocs/De1337/nothing/index.php on line 619

Warning: Cannot modify header information - headers already sent by (output started at /hermes/walnacweb04/walnacweb04ab/b2791/pow.jasaeld/htdocs/De1337/nothing/index.php:99) in /hermes/walnacweb04/walnacweb04ab/b2791/pow.jasaeld/htdocs/De1337/nothing/index.php on line 1169

Warning: Cannot modify header information - headers already sent by (output started at /hermes/walnacweb04/walnacweb04ab/b2791/pow.jasaeld/htdocs/De1337/nothing/index.php:99) in /hermes/walnacweb04/walnacweb04ab/b2791/pow.jasaeld/htdocs/De1337/nothing/index.php on line 1176

Warning: Cannot modify header information - headers already sent by (output started at /hermes/walnacweb04/walnacweb04ab/b2791/pow.jasaeld/htdocs/De1337/nothing/index.php:99) in /hermes/walnacweb04/walnacweb04ab/b2791/pow.jasaeld/htdocs/De1337/nothing/index.php on line 1176

Warning: Cannot modify header information - headers already sent by (output started at /hermes/walnacweb04/walnacweb04ab/b2791/pow.jasaeld/htdocs/De1337/nothing/index.php:99) in /hermes/walnacweb04/walnacweb04ab/b2791/pow.jasaeld/htdocs/De1337/nothing/index.php on line 1176

Warning: Cannot modify header information - headers already sent by (output started at /hermes/walnacweb04/walnacweb04ab/b2791/pow.jasaeld/htdocs/De1337/nothing/index.php:99) in /hermes/walnacweb04/walnacweb04ab/b2791/pow.jasaeld/htdocs/De1337/nothing/index.php on line 1176

Warning: Cannot modify header information - headers already sent by (output started at /hermes/walnacweb04/walnacweb04ab/b2791/pow.jasaeld/htdocs/De1337/nothing/index.php:99) in /hermes/walnacweb04/walnacweb04ab/b2791/pow.jasaeld/htdocs/De1337/nothing/index.php on line 1176

Warning: Cannot modify header information - headers already sent by (output started at /hermes/walnacweb04/walnacweb04ab/b2791/pow.jasaeld/htdocs/De1337/nothing/index.php:99) in /hermes/walnacweb04/walnacweb04ab/b2791/pow.jasaeld/htdocs/De1337/nothing/index.php on line 1176

Warning: Cannot modify header information - headers already sent by (output started at /hermes/walnacweb04/walnacweb04ab/b2791/pow.jasaeld/htdocs/De1337/nothing/index.php:99) in /hermes/walnacweb04/walnacweb04ab/b2791/pow.jasaeld/htdocs/De1337/nothing/index.php on line 1176

Warning: Cannot modify header information - headers already sent by (output started at /hermes/walnacweb04/walnacweb04ab/b2791/pow.jasaeld/htdocs/De1337/nothing/index.php:99) in /hermes/walnacweb04/walnacweb04ab/b2791/pow.jasaeld/htdocs/De1337/nothing/index.php on line 1176

Warning: Cannot modify header information - headers already sent by (output started at /hermes/walnacweb04/walnacweb04ab/b2791/pow.jasaeld/htdocs/De1337/nothing/index.php:99) in /hermes/walnacweb04/walnacweb04ab/b2791/pow.jasaeld/htdocs/De1337/nothing/index.php on line 1176

Warning: Cannot modify header information - headers already sent by (output started at /hermes/walnacweb04/walnacweb04ab/b2791/pow.jasaeld/htdocs/De1337/nothing/index.php:99) in /hermes/walnacweb04/walnacweb04ab/b2791/pow.jasaeld/htdocs/De1337/nothing/index.php on line 1176

Warning: Cannot modify header information - headers already sent by (output started at /hermes/walnacweb04/walnacweb04ab/b2791/pow.jasaeld/htdocs/De1337/nothing/index.php:99) in /hermes/walnacweb04/walnacweb04ab/b2791/pow.jasaeld/htdocs/De1337/nothing/index.php on line 1176

Warning: Cannot modify header information - headers already sent by (output started at /hermes/walnacweb04/walnacweb04ab/b2791/pow.jasaeld/htdocs/De1337/nothing/index.php:99) in /hermes/walnacweb04/walnacweb04ab/b2791/pow.jasaeld/htdocs/De1337/nothing/index.php on line 1176

Warning: Cannot modify header information - headers already sent by (output started at /hermes/walnacweb04/walnacweb04ab/b2791/pow.jasaeld/htdocs/De1337/nothing/index.php:99) in /hermes/walnacweb04/walnacweb04ab/b2791/pow.jasaeld/htdocs/De1337/nothing/index.php on line 1176

Warning: Cannot modify header information - headers already sent by (output started at /hermes/walnacweb04/walnacweb04ab/b2791/pow.jasaeld/htdocs/De1337/nothing/index.php:99) in /hermes/walnacweb04/walnacweb04ab/b2791/pow.jasaeld/htdocs/De1337/nothing/index.php on line 1176

Warning: Cannot modify header information - headers already sent by (output started at /hermes/walnacweb04/walnacweb04ab/b2791/pow.jasaeld/htdocs/De1337/nothing/index.php:99) in /hermes/walnacweb04/walnacweb04ab/b2791/pow.jasaeld/htdocs/De1337/nothing/index.php on line 1176

Warning: Cannot modify header information - headers already sent by (output started at /hermes/walnacweb04/walnacweb04ab/b2791/pow.jasaeld/htdocs/De1337/nothing/index.php:99) in /hermes/walnacweb04/walnacweb04ab/b2791/pow.jasaeld/htdocs/De1337/nothing/index.php on line 1176

Warning: Cannot modify header information - headers already sent by (output started at /hermes/walnacweb04/walnacweb04ab/b2791/pow.jasaeld/htdocs/De1337/nothing/index.php:99) in /hermes/walnacweb04/walnacweb04ab/b2791/pow.jasaeld/htdocs/De1337/nothing/index.php on line 1176

Warning: Cannot modify header information - headers already sent by (output started at /hermes/walnacweb04/walnacweb04ab/b2791/pow.jasaeld/htdocs/De1337/nothing/index.php:99) in /hermes/walnacweb04/walnacweb04ab/b2791/pow.jasaeld/htdocs/De1337/nothing/index.php on line 1176

Warning: Cannot modify header information - headers already sent by (output started at /hermes/walnacweb04/walnacweb04ab/b2791/pow.jasaeld/htdocs/De1337/nothing/index.php:99) in /hermes/walnacweb04/walnacweb04ab/b2791/pow.jasaeld/htdocs/De1337/nothing/index.php on line 1176

Warning: Cannot modify header information - headers already sent by (output started at /hermes/walnacweb04/walnacweb04ab/b2791/pow.jasaeld/htdocs/De1337/nothing/index.php:99) in /hermes/walnacweb04/walnacweb04ab/b2791/pow.jasaeld/htdocs/De1337/nothing/index.php on line 1176
8000 GitHub - dszczyt/repligrate
Nothing Special   »   [go: up one dir, main page]

Skip to content

dszczyt/repligrate

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

5 Commits
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

Repligrate

Repligrate is a tool that automatically generates pgroll migrations by listening to PostgreSQL schema changes via logical replication. It acts as a logical replica to capture DDL (Data Definition Language) operations and converts them into pgroll-compatible migration files.

Features

  • 🔍 Automatic Schema Change Detection: Listens to PostgreSQL logical replication stream to detect schema changes
  • 📝 pgroll Migration Generation: Automatically converts detected schema changes into pgroll migration JSON format
  • 🔄 Logical Replication: Uses PostgreSQL's native logical replication mechanism for reliable change capture
  • 🛡️ Zero-Downtime Ready: Generated migrations are compatible with pgroll's zero-downtime migration approach
  • ⏸️ Pause/Continue: Temporarily pause change detection without stopping the application
  • 🔧 Constraint Support: Full support for ADD/DROP CONSTRAINT operations (PRIMARY KEY, UNIQUE, FOREIGN KEY, etc.)
  • Migration Optimization: Automatically merges related schema changes to reduce migration count
  • 🎯 Selective Monitoring: Filter which schemas and tables to monitor
  • 📦 Easy Integration: Simple CLI interface for integration into CI/CD pipelines

Architecture

PostgreSQL Database
        ↓
   Logical Replication
        ↓
   WAL Message Parser
        ↓
   Schema Change Detector
        ↓
   pgroll Migration Generator
        ↓
   Migration Files (JSON)

Installation

From Source

git clone https://github.com/dszczyt/repligrate.git
cd repligrate
cargo build --release

The binary will be available at target/release/repligrate.

From GitHub Releases

Download pre-built binaries from the releases page:

# Linux (amd64)
curl -LO https://github.com/dszczyt/repligrate/releases/latest/download/repligrate.linux.amd64
chmod +x repligrate.linux.amd64
sudo mv repligrate.linux.amd64 /usr/local/bin/repligrate

# macOS (Apple Silicon)
curl -LO https://github.com/dszczyt/repligrate/releases/latest/download/repligrate.macos.arm64
chmod +x repligrate.macos.arm64
sudo mv repligrate.macos.arm64 /usr/local/bin/repligrate

# Windows (amd64)
# Download from: https://github.com/dszczyt/repligrate/releases/latest/download/repligrate.win.amd64.exe

Quick Start

1. Initialize Replication

export DATABASE_URL="postgres://user:password@localhost:5432/mydb"
repligrate listen --slot-name repligrate_slot --publication-name repligrate_pub

2. Make Schema Changes

In another terminal, make schema changes to your database:

CREATE TABLE users (
    id SERIAL PRIMARY KEY,
    name VARCHAR(255) NOT NULL,
    email VARCHAR(255) UNIQUE
);

ALTER TABLE users ADD COLUMN created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP;

3. Generated Migrations

Repligrate will automatically generate pgroll migration files in the output directory:

{
  "name": "migration_20240101_120000",
  "operations": [
    {
      "create_table": {
        "name": "users",
        "columns": [
          {
            "name": "id",
            "type": "integer",
            "pk": true
          },
          {
            "name": "name",
            "type": "varchar(255)",
            "nullable": false
          }
        ]
      }
    }
  ]
}

Configuration

Environment Variables

  • DATABASE_URL: PostgreSQL connection string (required)
  • OUTPUT_DIR: Directory for generated migrations (default: ./migrations)

Configuration File

Create a repligrate.toml:

database_url = "postgres://user:password@localhost:5432/mydb"
output_dir = "./migrations"

[replication]
slot_name = "repligrate_slot"
publication_name = "repligrate_pub"
batch_size = 1000
receive_timeout = 30

[schema_filter]
include_schemas = []
exclude_schemas = ["pg_catalog", "information_schema"]
include_tables = []
exclude_tables = []

CLI Commands

listen

Start listening for schema changes:

repligrate listen \
  --slot-name repligrate_slot \
  --publication-name repligrate_pub \
  --tables "public.users,public.orders"

Options:

  • --slot-name: Name of the replication slot (default: repligrate_slot)
  • --publication-name: Name of the publication (default: repligrate_pub)
  • --tables: Comma-separated list of tables to monitor (empty = all)

status

Check replication status:

repligrate status

pause

Pause the replication listener:

repligrate pause --slot-name repligrate_slot

This temporarily stops processing schema changes without stopping the application. Useful for maintenance windows or reducing load.

continue

Continue the replication listener:

repligrate continue --slot-name repligrate_slot

Resume processing schema changes after a pause.

cleanup

Clean up replication resources:

repligrate cleanup --slot-name repligrate_slot

test

Generate a test migration:

repligrate test

Supported Schema Changes

Repligrate currently supports the following schema change operations:

  • ✅ CREATE TABLE
  • ✅ DROP TABLE
  • ✅ ALTER TABLE
  • ✅ ADD COLUMN
  • ✅ DROP COLUMN
  • ✅ MODIFY COLUMN
  • ✅ ADD CONSTRAINT (PRIMARY KEY, UNIQUE, FOREIGN KEY, CHECK)
  • ✅ DROP CONSTRAINT
  • ✅ CREATE INDEX
  • ✅ DROP INDEX

Migration Optimization

Repligrate automatically optimizes migrations by merging related schema changes:

  • Consecutive ADD COLUMN operations are merged into a single ALTER TABLE statement
  • Consecutive ADD CONSTRAINT operations are merged into a single ALTER TABLE statement
  • Consecutive ALTER TABLE operations are merged when possible
  • Rollback SQL is automatically generated for constraint operations

How It Works

  1. Replication Slot Creation: Creates a logical replication slot to capture changes
  2. Publication Setup: Creates a publication for the tables to monitor
  3. WAL Monitoring: Listens to the Write-Ahead Log (WAL) stream
  4. DDL Detection: Filters for DDL operations (schema changes)
  5. Migration Generation: Converts DDL operations to pgroll migration format
  6. File Output: Writes migrations to JSON files with timestamps

Requirements

  • PostgreSQL 14.0 or later
  • Rust 1.70 or later (for building from source)
  • wal_level = logical in PostgreSQL configuration

PostgreSQL Configuration

Ensure your PostgreSQL instance is configured for logical replication:

-- Check current settings
SHOW wal_level;  -- Should be 'logical'
SHOW max_wal_senders;  -- Should be > 0
SHOW max_replication_slots;  -- Should be > 0

If needed, update postgresql.conf:

wal_level = logical
max_wal_senders = 10
max_replication_slots = 10

Then restart PostgreSQL.

Development

Running Tests

cargo test

Building Documentation

cargo doc --open

Limitations

  • Currently supports basic DDL operations
  • Complex migrations may require manual adjustment
  • Requires PostgreSQL 14.0+
  • Logical replication must be enabled

Versioning

Repligrate follows Semantic Versioning (SemVer), the same as pgroll.

  • Current version: v0.1.0
  • Version format: v{MAJOR}.{MINOR}.{PATCH}
  • See VERSIONING.md for detailed versioning guidelines

Contributing

Contributions are welcome! Please feel free to submit a Pull Request.

License

This project is licensed under the MIT License - see the LICENSE file for details.

Related Projects

Support

For issues, questions, or suggestions, please open an issue on GitHub.

About

No description, website, or topics provided.

Resources

Stars

Watchers

Forks

Packages

No packages published

Languages

0