Sylvain Kalache

Can every Kafka topic become an Iceberg table: instantly, zero‑copy, and 100 % open‑source? At Aiven we think so. A native “Iceberg Topic” could erase connector tax, end lock‑in, and make *topic = table* the new default. We’ve just blogged about the roadblocks we see today and the path forward to build together. 🚀 #ApacheKafka #ApacheIceberg #OpenSource #DataStreaming https://lnkd.in/eptGbBTf

165

12 Comments

Core Integration Patterns for Modern Systems 🔹 Peer-to-Peer Enables direct communication between two components without requiring a central coordinator. 🔹 API Gateway Serves as a single entry point for all client requests to backend services. 🔹 Pub-Sub Decouples message producers from consumers through a message broker for flexible event-driven architectures. 🔹 Request-Response The foundational integration pattern where a client sends a request to a server and waits for a response. 🔹 Event Sourcing Stores application state changes as an immutable sequence of events. Provides a complete audit trail. 🔹 ETL Extracts data from multiple sources, transforms it into a structured format, and loads it into a destination database for analysis and reporting. 🔹 Batching Accumulates data over time or until reaching a specified threshold before processing it as a single group. 🔹 Stream Processing Enables continuous ingestion, processing, and analysis of data streams in real-time for immediate insights and actions. 🔹 Orchestration Uses a central coordinator to manage interactions between distributed components or services in a workflow. -- Subscribe to our weekly newsletter to get a Free System Design PDF (158 pages): https://bit.ly/496keA7 #systemdesign #coding #interviewtips

981

12 Comments

Aspire + Coding Agents: 27 authors → 109 commits → 41 PRs in a week. What we learned: - Code reviews are critical. AI slop slips in. - Tests aren’t enough - humans still judge experience. - Everyone can contribute - even PMs are using agents for fixes. - Automate everything you can. - Make every PR shippable - make it as easy as possible to experiment. 💡 Humans are UI threads. Agents are background threads. Protect your UI threads (props to Mitch Denny) The bottleneck isn’t writing code anymore - it’s scaling review, validation, and quality at AI speed. #github #codingagent #aspire

536

17 Comments

Migrations are the least fun - and most challenging - part of scaling. Scaling in the age of AI isn’t just about running a lean team - it’s also about tackling migrations, managing compute costs, and building products that can handle millions of users without breaking. Michael T. (of AnySphere/Cursor fame) and Parker Conrad (of Rippling fame 😃) recently sat down to discuss a wide range of topics: ⚙️ Why migrations are the hardest (and least fun) part of scaling ⚙️ How AI changes the calculus of team size and productivity ⚙️ The trade-offs between speed, distribution, and long-term margin ⚙️ Building technical moats in markets moving at AI speed 👉 If you’re building in AI right now, or thinking of building something soon, give this a watch! https://lnkd.in/ggpMtcyG

255

7 Comments

Accidentally discovered a new way to work with GitHub Copilot today: write new tests based off test coverage reports. Copilot (or I) run this command to generate a report of all the uncovered lines in the current diff: pytest --cov --cov-report=xml && \ diff-cover coverage.xml --html-report coverage_report.html && \ open coverage_report.html Then Copilot reviews the report, summarizes the uncovered situations, and proposes new test cases to add, or if the code is truly unreachable, suggests deleting it. Just another reason why you should add diff-cover (or similar tool) to your projects!

115

4 Comments

The snapshot command in DDEV lets you save your database state and restore it later. Think of it as a checkpoint system for your development environment. Here's how to use it: - Create a snapshot: "ddev snapshot --name=snapshot_name" - List your snapshots: "ddev snapshot --list" - Restore a snapshot: "ddev snapshot restore snapshot_name" - Restore latest snapshot: "ddev snapshot restore" I use this constantly during live streams and tutorials. Set up your site exactly how you want it, take a snapshot, then experiment freely. When things break, just restore back to your clean state. #ddev ~~~ If you found this post helpful. ♻️ Please repost and follow.

52

5 Comments