1020 articles curated by AI from 15+ sources. Updated every 6 hours.
This article, part two of a series, explores different forms of data leakage beyond temporal leakage, including spatial, structural, and coverage-related issues. It discusses how these subtle leakages can deceptively inflate model performance.
This article from Meta Engineering outlines the architectural blueprint for their AI storage systems, designed to handle the exponential growth of model capabilities and training dataset sizes. It details the importance of reliable and fast storage access for rapid AI development and computation.
This article details the architectural journey and engineering challenges faced by OpenAI in delivering low-latency voice AI services to 900 million users. It explores the system design choices and optimizations necessary to achieve performance at such a massive scale.
This article introduces Inductive Latent Context Persistence (ILCP) as a method to optimize multi-hop LLM agent pipelines. It explains how ILCP reduces the cost of hand-offs by transferring a compressed hidden state, allowing downstream agents to avoid re-creating context and mitigating the agent co
This article details benchmarks of the One Billion Row Challenge (1BRC) conducted on a Threadripper 9980X processor. It compares these new results to the original benchmarks run on an EPYC 7502P, focusing on the performance of processing large datasets.
The article announces chDB-WASM, a project that compiles the entire ClickHouse OLAP engine to WebAssembly. This development enables browser-native SQL and embedded analytics applications. It represents a significant technical achievement in making powerful analytical databases accessible in edge env
This 'Show HN' presents SyntheticRows, a project designed to expand small datasets by generating synthetic data. The tool includes a feature for providing an 'honest quality score' alongside the expanded datasets.
This article explores strategies for managing memory bottlenecks in data engineering workflows, particularly when scaling compute resources is not feasible. It details how techniques such as Pandas chunking and the use of Dask and Polars can help process millions of records effectively.
The article explores the strategic placement of AI agents within data engineering pipelines, proposing their role in a dedicated "correctness layer." It discusses how these agents can enhance data quality and reliability. The post outlines conceptual frameworks for integrating agentic architectures
The article details the architectural considerations and design principles for building a Multi-Agent Collaboration Protocol (MCP) server to manage unstructured data. It explores challenges specific to unstructured data and proposes solutions for agent communication and data handling. The post provi
This paper proposes DA-Studio, an agentic system aimed at automating multi-step data analysis workflows from heterogeneous inputs. The system focuses on autonomously organizing tasks and executing generated code within a controlled environment. Its design addresses the complexity of real-world data
This research introduces a substantial dataset of real-world addresses intended for benchmarking data cleaning processes. It aims to overcome limitations of existing controlled-environment benchmarks by providing a more representative collection for evaluating error detection and correction methods
This research focuses on improving the reliability of large language models for structured reasoning tasks like Text-to-SQL at inference time. It proposes a new approach called Outcome Reward Models for test-time verification, departing from traditional methods like Best-of-N sampling or Majority Vo
This paper highlights the prohibitive costs associated with using proprietary large language model APIs for operations on massive databases. It contends that LM-enhanced relational operators can incur significant expenses, potentially exceeding $10,000 for a single query. The authors advocate for th
This paper investigates the accuracy challenges faced by LLM-based agents in industrial asset operations when reasoning over flat document stores. It references AssetOpsBench, which shows GPT-4 agents achieving only 65% accuracy in certain scenarios. The authors argue for integrating knowledge graph
This paper tackles the fundamental challenge of identifying linear utility functions that align with observed candidate rankings in various applications. The research has implications for fields such as admissions, hiring processes, and recommendation systems. It builds upon previous work concerning
The article details how ClickHouse Managed Postgres scales PgBouncer beyond its single-threaded limitation. It describes running a peered fleet of PgBouncer processes utilizing so_reuseport to distribute connection pooling across multiple CPU cores. Benchmarks demonstrate the effectiveness of this a
This article explores multi-token residual prediction, a specialized topic within machine learning. It likely delves into the mathematical or algorithmic details of this prediction method, potentially for optimizing model performance or efficiency.
The article investigates why AI agent loops frequently fail in production environments, attributing these failures primarily to issues with managing agent state across iterations rather than model performance. It then proposes specific database patterns designed to address these state management cha
A Postgres contributor and SQL standards committee member reports on the outcomes of the latest SQL standardization meeting. The article details new SQL standard features, including QUALIFY and INSERT ... BY NAME, and discusses their specific implications for the Postgres database.
Artemis Security achieved a 69x reduction in detection query times and up to 60x faster investigative lookups by optimizing its ClickHouse deployment. The article highlights the use of ClickHouse query coalescing, materialized extraction, and AI-powered debugging with Claude for these performance ga
This article introduces ScarfBench, a benchmark specifically designed to evaluate AI agents tasked with migrating enterprise Java frameworks. It details the methodology for assessing agent performance in this complex domain.
This article discusses the evolution of database architectures, moving from traditional monoliths to the Lakebase concept and ultimately to LTAP. It explores a re-evaluation of database design starting from the underlying storage layer, providing insights into future database paradigms.
This article describes how to enable AI agents to record video demonstrations of their operations and output using the `shot-scraper video` tool, offering a method for evaluating or showcasing agent performance.
The article details the concept of context engineering within RAG pipelines, identifying four distinct typed inputs that feed into an LLM call to generate a RAG answer. It also mentions follow-up work on corpus, conversation, and tool extensions.
This article examines a research preview, specifically focusing on the concept of an interaction model as proposed by Thinking Machines. It delves into the details of what this model entails and its implications for system design.
This article presents a detailed benchmark of Hardwood 1.0, a Java library for reading Parquet files, executed on a Threadripper 9980X. It compares Hardwood's row and columnar reader APIs against initial benchmarks published by the author, Gunnar Morling, in the v1.0 announcement.
The article presents a field guide to hybrid local-cloud LLM patterns, offering a hands-on walkthrough of a workflow combining Gemma 4 and GPT-5.4. It covers aspects like reasoning and structured outputs within this hybrid approach.
OpenCode integrates with Supabase, allowing an agent to connect to databases, Edge Functions, and logs. The article explains how MCP setup is configured automatically for this integration.
This research explores a new method for database context compression, specifically designed to improve Text-to-SQL performance on large, complex enterprise databases. The paper argues that current Text-to-SQL models struggle with real-world benchmarks like Spider 2.0 and BIRD, proposing a solution t
This paper investigates algebraic subgraph counting, a core problem within graph analytics that involves determining the number of subgraph isomorphisms for a query graph within a larger data graph. It builds upon the candidate tree-based framework, offering insights into its application for efficie
This research presents CADENZA, a system designed to transform natural language intent into task-specific operator Directed Acyclic Graphs (DAGs) for semantic query processing engines. It details how these engines extend relational query processing with semantic operators executed via model inferenc
This paper offers a comprehensive comparative analysis of established enterprise data modeling methodologies, including Inmon, Kimball, and Data Vault. It discusses their design principles and long-term impacts on analytical capabilities, operational agility, and regulatory compliance for data-drive
This research introduces SemJoin, a method for optimizing semantic joins to integrate unstructured data into relational database systems. It focuses on evaluating joins under natural-language predicates, leveraging large language models to enhance natural language querying and analysis capabilities.
This paper describes Mandol, an agglomerative agent memory system developed to support long-term conversations in AI agents. It tackles the complexities of remembering and querying cross-session, multi-typed information with intricate correlations, contrasting with existing systems that often rely o
This research argues for a new class of database system architectures to support emerging long-horizon agentic tasks such as code generation and scientific discovery. It proposes Experience Graphs as a foundational data structure for building self-improving agents, recognizing the need for systems t
This paper introduces CLIP, a lightweight pruning method for inverted file (IVF)-based vector search, a key component in modern multimodal retrieval systems. It leverages cosine-law principles to enhance the scalability, update efficiency, and hardware friendliness of these widely adopted vector sea
This paper presents MaDI-Bench, an end-to-end benchmark designed for evaluating comprehensive data integration processes. It covers a sequence of interdependent tasks including schema matching, value normalization, entity blocking, entity matching, and data fusion, aiming to provide a coherent repre
The article presents GRAB, a constructor-encoder-bridge pipeline for multi-table question answering. This method converts relational data into a heterogeneous graph, encodes it using message passing, and then transfers these signals to a Large Language Model via a set of query-contextualized latent
The article posits that tabular foundation models cannot effectively reason about data produced by running systems without explicit access to the rules governing those systems. It formalizes this limitation and introduces the Operational Turing Test, which constructs pairs of legal and rule-violatin
The article introduces SAKE, a Software Architectural Knowledge Evaluation benchmark, to measure Large Language Models' reasoning abilities in software architecture. It highlights that while LLMs assist in software development, their competence in architectural decision-making, which relies on under
The article investigates the performance of on-premises, open-weight Large Language Models for Text-to-SQL tasks. It offers a fully reproducible benchmark on the BIRD dataset, assessing how well these models perform and which popular accuracy-improving techniques provide worthwhile compute returns,
The article proposes an elastic scheduling mechanism for intermittent query processing in a cluster environment. It addresses applications that process tuple streams over a window, requiring results by a deadline, by processing tuples in batches rather than continuously to balance resource usage and
This article details how Redpanda's Cloud Topics architecture rethinks Kafka compaction to overcome common issues like disk saturation and CPU overload in traditional Kafka clusters. It explains the redesign's approach to reducing redundant work, lowering cloud storage expenses, and preserving Kafka
OpenAI engineers conducted large-scale core dump analysis to diagnose and resolve rare infrastructure crashes. This investigation led to the discovery and rectification of both a hardware fault and a software bug that had persisted for 18 years.
This article details the architecture and methodology behind Genebench-Pro, a new benchmark designed to evaluate advanced AI agent capabilities in complex reasoning tasks. It covers the system's components, evaluation metrics, and initial findings regarding agent performance.
Jua replaced a file-based forecast pipeline with ClickHouse Cloud, reducing data delivery time from one hour to 20 minutes and historical query times from hours to seconds. This improved speed provides an advantage for energy traders.
The ClickHouse team scaled their internal logging platform, LogHouse, from 19 PiB to 431 PiB and 1.59 quadrillion rows across three cloud providers. The article details how they rearchitected the system to manage 80 GiB/s of writes while maintaining fast queries and minimizing underlying complexity.
This article details the enhancements introduced in Ray Data version 2.56, with a specific focus on improving reliability for AI data pipelines. It covers features designed to create more robust and fault-tolerant data processing workflows for machine learning applications.
Discord's API spans over 1700 endpoints across hundreds of Kubernetes deployments. The article describes the challenge of accurately tracking per-feature hosting costs without requiring extensive system restructuring. Jim Benton explains the methodologies and approaches Discord adopted to address th
This article explores methods and considerations for extracting structured data from HTML tables using AI agents. It likely details challenges in parsing varied HTML structures and how agents can be configured or prompted to accurately identify and extract tabular information.
This article argues that a data-platform-centric approach, rather than solely focusing on smarter models, is crucial for developing legal AI that is governed, context-aware, and contributes to institutional intelligence. It highlights the importance of the data platform in shaping AI outcomes.
This article describes a specific application of AI agents to interact with a user's operating system, detailing how an agent can be configured to count the number of open tabs in the Safari browser. It likely delves into the mechanics of agent-system integration and tool invocation.
This article introduces DiScoFormer, a novel transformer architecture designed to jointly model density and score functions across various data distributions. It details the design and capabilities of this unified model.
The article introduces Ornith-1.0, a framework that enables Large Language Models to self-scaffold for agentic coding tasks. It details how LLMs can generate and refine their own execution plans and tools to improve coding performance and reliability.
This article explains the architectural patterns behind how AI agents manage memory and prevent forgetfulness. It explores the foundational constraints driving these designs, examines the resulting system architectures, and discusses the associated tradeoffs.
This article highlights how minor prompt changes can silently disrupt critical LLM behavior in production, introducing a practical framework designed to detect these hidden prompt regressions before they impact users.
The article explores GenPage, a system developed at Netflix for end-to-end generative homepage construction. It likely delves into the architecture and technical considerations behind building such a system for a large-scale platform.
The article explores the rising trend of small language models and provides guidance on how to choose between these smaller models and larger frontier models for various applications.
This article argues that benchmarks are only meaningful if they are inspectable, challengeable, and reproducible. It details the findings from testing Databricks' claims regarding ClickHouse performance and highlights discrepancies in methodology.
This article explores the paradigm shift required in software and hardware design when the primary user becomes an LLM rather than a human. It discusses early considerations for building experiences optimized for AI agents.
This article discusses key architectural considerations for choosing PostgreSQL-compatible databases to support AI at scale. It covers various evaluation criteria that impact long-term system viability and operational costs for projects needing robust database foundations.
The article explains the reliance of AI agents on search for world understanding before action. It details how Materialize assists in keeping computed search documents and vector embeddings current and synchronized with changes in underlying source systems.
The article explores the engineering challenges of building reliable agentic workflows for customer-facing APIs. It highlights that consistent delivery is a problem of variance, not just speed, and proposes counterintuitive solutions to ensure timely and usable high-quality answers.
This article provides a comparison of Retrieval Augmented Generation (RAG) approaches, specifically contrasting standard RAG with Graph RAG and Agentic RAG. It outlines the three distinct methods for connecting Large Language Models to data.
This article describes a team's experience building an AI inference routing layer to reduce costs, which initially halved their AI bill but led to a decline in customer satisfaction. It identifies cost-optimization routing layers as a Pareto trap and presents a methodology for detecting such quality
The article presents Framesmith 1.7, an open-source project that functions as a quality gate for AI agents. This system informs an AI agent when a user interface has reached a 'done' state, providing a mechanism for agents to assess UI completion.
This article provides a critical assessment of MySQL's recently introduced governance model. It discusses the perceived advancements and setbacks of these changes, offering insights into the implications for the database's future development and community involvement.
This article recounts the outcomes after 2,000 individuals attempted to exploit vulnerabilities in an AI assistant. It likely covers security challenges, adversarial interactions, and insights gained from real-world testing of AI agent robustness.
This article presents an incident report detailing a security vulnerability, identified as CVE-2026-LGTM. It likely provides a technical analysis of the issue and lessons learned from its resolution.
This article explores methods and considerations for reliably extracting and attributing specific information from OpenAI models. It focuses on techniques to ensure agents can accurately quote or reference model outputs, addressing challenges in building dependable AI agent workflows.
The article links to the GitHub repository for the DuckDB SQLite Extension. This extension allows users to integrate DuckDB's analytical capabilities with existing SQLite databases.
The article advocates for using PostgreSQL as a task queue solution. It then details strategies and methods for scaling Postgres queues to handle increased loads and maintain performance in production environments.
This article details the process of constructing a lightweight research agent. It demonstrates using Gemma 4, Ollama for local LLM inference, the OpenAI Agents SDK, and Tavily MCP for tool integration.
This article reports on Oracle's commitments to open up MySQL's governance model and the community's demand for concrete guarantees regarding these changes. It covers the ongoing dialogue between Oracle and the MySQL community concerning future development and control.
This article explores the concept of overfitting within the context of RAG evaluation. It discusses why achieving high scores on evaluation benchmarks does not necessarily equate to a comprehensive understanding of the subject matter by the RAG system.
The article details the author's changed perspective on AI after a year focused on security and governance. It likely explores practical challenges, risk management, and ethical considerations encountered when deploying AI in real-world scenarios.
This article from Fig.inc discusses challenges in improving browser-use models beyond simply increasing training data. It explores inherent issues and limitations in current data collection and model design for complex user interactions. The post provides insights into diagnosing and addressing pers
This article presents an architectural philosophy for building Retrieval-Augmented Generation (RAG) systems in enterprise environments. It discusses the strategic choices and design principles necessary for integrating expert knowledge into RAG pipelines. The post provides a framework for robust and
This post explores the concept of 'bounded cognition' in engineering, advocating for system designs that account for human cognitive limits. It discusses strategies to reduce complexity and improve comprehensibility in software and data systems. The article offers a framework for building more manag
This article explores the evolution of context engineering in AI, moving from simply maximizing token input to deliberate data curation for LLMs. It discusses the limitations of large context windows and the need for structured, relevant information to improve AI model performance. The post outlines
This project introduces Loomabase, an open-source solution enabling column-level Conflict-free Replicated Data Type (CRDT) synchronization for SQLite and PostgreSQL databases. It allows for robust, eventually consistent replication of individual column changes across distributed environments. The re
This resource introduces Monedula, a simulator designed for Apache Kafka environments. It allows users to model and test Kafka cluster behavior under various load conditions and configurations. The simulator aids in understanding Kafka internals and optimizing streaming data architectures without de
This article presents an engineering handbook focused on patterns for constructing software systems that handle money. It aims to provide architectural insights and practical guidance relevant to fintech development.
This article argues that context loss is the primary factor slowing down engineering teams utilizing AI for coding tasks. It explores how the inability of AI tools to maintain sufficient context impacts developer efficiency.
This article examines query cost model calibration for databases deployed within confidential virtual machines, specifically noting AMD SEV-SNP. It addresses the protection of sensitive cloud data while minimizing changes to legacy database management systems.
This article introduces 3D spatial pattern matching, defining it as the process of aligning query entities and constraints with database entities and relations. It covers various applications, including similar region search and road network matching.
This article proposes EcoTable, a method for cost-effective table integration within data lakes, designed to overcome challenges posed by diverse formats like CSV and Parquet. It also facilitates data access through natural language queries, aiming to simplify traditional ETL.
This article introduces BtrLog, a system designed to provide low-latency write-ahead logging for cloud database systems. It addresses the challenges of achieving WAL durability with remote storage, specifically mentioning the latency issues associated with options like EBS.
Entity Matching (EM) is a fundamental operation in data integration pipelines, focused on comparing records from different sources to determine if they refer to the same real-world entity. This paper introduces a method that incorporates domain information and addresses distribution alignment within
This article discusses the importance of SQL grammar in a query engine like Trino. It explains how SQL is defined by its grammar, including predicates, operators, and forms. The post likely covers Trino's adherence to the ISO 9075 standard and ongoing work on its SQL dialect.
An experienced software engineer details their method for identifying vulnerabilities within the ClickHouse codebase using large language models such as GitHub Copilot, Claude Opus, and Gemini. The process involves generating hypotheses and accelerating the validation of potential security flaws wit
The article details how to deploy a vLLM server on Hugging Face Jobs using a single command. It covers the setup and operational steps necessary for running high-performance LLM inference in a hosted environment.
The article from Meta Engineering discusses the necessity of robust data understanding for effective privacy controls in AI-native environments. It presents a case study on asset classification, detailing how systems must precisely identify data to enforce policies like retention, access, and anonym
The article introduces Hardwood 1.0, a new open-source Apache Parquet reader for the JVM optimized for speed and minimal dependencies. It discusses the design decisions made to enhance performance and reduce the memory footprint for processing Parquet files.
The video discusses various data benchmarks and their inherent limitations. It explores the methodologies used in data benchmarking and highlights common pitfalls and considerations when interpreting performance metrics.
The article introduces Topos, a system addressing the challenge of reviewing code generated rapidly by AI agents. Topos parses programs into various graph representations, such as AST and CFG, to score them across structural quality pillars including simplicity, composability, and security.
The article explores the technical advantages of Parquet, moving beyond its common perception as a simple CSV alternative. It details how Parquet's columnar format, compression, and schema evolution capabilities offer significant performance benefits for data storage and processing.
The article introduces a context graph layer built to augment multi-agent memory, moving beyond limitations found in traditional vector RAG approaches. It includes a benchmark comparing raw chat history, vector-only RAG, and this new context graph layer, revealing insights into relational retrieval
The article details the complex challenges involved in testing a Kafka proxy, specifically addressing the need to manage millions of permutation test cases. It describes the engineering approaches and methodologies developed to ensure robust and reliable proxy behavior in a streaming environment.
The article presents a reproducible benchmark for payment fraud detection, evaluating the performance of Gradient Boosted Decision Trees (GBDTs) versus AI agents. It analyzes their efficacy across metrics like latency, cost, and reproducibility, delineating scenarios where each technology proves mor
The article introduces "Autodata," an agentic data scientist designed for generating high-quality synthetic data. It describes the agent's architecture and methodologies for creating synthetic datasets that maintain fidelity and utility for various analytical tasks.
The article introduces WAL-RUS, a rewrite of the WAL-G tool for Postgres backups, now implemented in Rust. It covers the technical rationale behind the rewrite and the features of this new open-source project aimed at improving backup reliability and performance.
This project, BrainAPI, constructs event-centric knowledge graphs from various data sources through a four-stage pipeline. The GitHub repository provides details on its architecture and implementation for transforming raw data into structured graph representations.
This article explores the performance of hybrid language models by analyzing which tokens they predict more accurately. It likely provides insights into the operational characteristics and architectural advantages of these models in various prediction scenarios.
Pinterest details the architectural and engineering strategies they implemented to achieve near-linear training scalability for their foundation models. The article describes the specific optimizations and distributed systems design choices made to efficiently train large-scale ML models within thei
Addresses the complexities of provisioning data access for natural language data queries within large-scale systems. The article explores methods and architectural patterns for managing access control and ensuring data security in such environments.
The article details methods for running three distinct LLMs concurrently on a single 8GB GPU, addressing common VRAM limitations. It explains the use of C++ layer multiplexing and admission control to manage resources and achieve parallel inference on bare metal hardware.
This article describes the process of creating a small, custom compiler designed to process data-parallel kernels. It delves into the architectural considerations and implementation specifics required to achieve optimized execution for data-intensive tasks.
This article introduces the Arbiter Pattern for RAG systems, where an LLM is employed to rank and select the most appropriate document from a set of candidates. It outlines how a single LLM call generates a reasoned choice, producing a structured output for auditing.
Cloudflare details the development of saga-style rollbacks for its Workflows durable execution engine, which handles multi-step applications. The article explains how developers can now define compensating actions for each step within a workflow to ensure transactional consistency.
This article details methodologies for constructing one-minute Open-High-Low-Close (OHLC) bars from market snapshot data that is not uniformly sampled. It outlines the technical steps involved in time-series aggregation from irregular inputs.
This article introduces MAVS-GC (Multi Adaptive Vetting Systems-Governance Core), an open-source project proposing a specific governance architecture for AI systems. The project investigates the impact of an explicit governance layer placed atop multiple specialist components on overall system behav
This article proposes treating the LLM context window as a data assembly problem, focusing on structuring and optimizing data input for large language models. It likely explores strategies for efficiently preparing machine-readable metadata and contextual memory to improve LLM performance.
This article presents TabClean, a method that employs LLM-synthesized programs to address common data cleaning challenges in tabular data. It focuses on resolving issues such as missing values, inconsistent formats, and violated dependencies frequently encountered in production analytics and machine
This article introduces CV-rules, a new method for characterizing and verifying serializability in concurrency control protocols. It proposes that a transaction order must satisfy specific per-read conditions, the C-rule (Causality) and V-rule (View Consistency), to constrain reads-from relationship
This arXiv paper introduces VADAOrchestra, a framework for neurosymbolic orchestration of adaptive reasoning workflows. It addresses how real-world decision-making dynamically evolves with new context and data, moving beyond traditional fixed business processes.
The article details a specific problem found in Apache Kafka's log compaction process that can lead to data corruption. It explains how to reproduce this issue and outlines the method Redpanda used to resolve it within their platform.
Details the routing mechanisms employed for serverless servers, focusing on the integration and functionality of Pingora, Envoy, and Spanner. The article explores how these components work together to manage traffic in a serverless environment.
This Weaviate 1.38 release introduces several key features, including the general availability of the HFresh disk-based vector index and the built-in MCP Server. It also details the re-engineered cluster-wide asynchronous replication, which now operates from a single scheduler, and previews the Boos
ClickHouse announces Silk, a new open-source C++ fiber runtime designed for its database, featuring a NUMA-aware work-stealing scheduler and io_uring I/O. The article highlights its zero heap allocation in the steady state, achieving nanosecond-level fiber yields and significantly reducing tail late
This article details how Vibe.co managed to scale its Connected TV ad impression data from 100 GB to 2 TB without requiring architectural changes. It specifically covers their migration process from Postgres to ClickHouse Cloud as the solution for handling billions of ad impressions.
This entry points to the `simonw/browser-compat-db` GitHub repository, a project by Simon Willison. The repository likely involves a database focused on browser compatibility, potentially integrating advanced LLM or agent technologies as indicated by its associated tags.
The article explores methods to assess the success of Kafka self-service platforms, focusing on key metrics and indicators. It details how organizations can determine if their self-service initiatives are truly empowering developers and streamlining operations.
This article details Pinterest's implementation of automated schema evolution within their next-generation database ingestion framework. It describes the system designed to manage and adapt to schema changes automatically at scale.
The article transitions from traditional coding practices to 'agentic engineering' by leveraging AI models like Claude Code. It discusses how AI agents can assist in code generation, debugging, and overall software development workflows.
The article outlines expected features and improvements for the upcoming PostgreSQL 19 release, currently in beta. It details advancements in areas like performance, new SQL functionalities, and potential architectural changes for the database.
The article discusses the development of a dedicated data infrastructure layer tailored for AI, focusing on how web-scale data can be effectively organized and served to AI models. It examines the architectural components and challenges involved in building these new data pipelines for AI consumptio
DBOSify is presented as an open-source project designed to replace Temporal-style workflow orchestration, leveraging PostgreSQL for durable state management. It aims to provide a reliable, ACID-compliant platform for building complex, long-running applications.
The article reports that medical diagnosis AI systems can be manipulated to reveal specific data points from their training datasets. This vulnerability allows for the identification of individuals whose medical information was used to train these models. The finding highlights a significant privacy
This article describes a technique to accelerate VLM (Vision Language Model) fine-tuning. It explains how LanceDB, Lance format, and Geneva are used to materialize expensive multimodal features once, allowing subsequent training directly from these pre-computed columns.
The article outlines a practical onboarding workflow for a new data engineer, emphasizing the immediate task of making ETL pipelines testable. It details steps for setting up environments, implementing automated testing protocols, and leveraging AI for development assistance. The guide focuses on es
The article presents findings on the current landscape of agentic analytics, synthesizing experiences from 50 real data teams. It covers the adoption, challenges, and evolving patterns in using AI agents for analytical tasks. The report offers a snapshot of how organizations are integrating agentic
The article details methods for accelerating the fine-tuning process of transformer models using NVIDIA NeMo AutoModel. It explores how this framework can optimize computational efficiency and reduce training times for large language models. The content focuses on practical techniques to enhance per
This article examines the constraints and tradeoffs of large versus small language models. It delves into three layers of model design and investigates production systems that combine both types of models.
The article investigates the internal mechanisms of factual recall within Gemma-2B and Gemma-12B-IT transformer models. It employs activation patching to reveal a three-phase circuit for how facts are stored, routed, and retrieved across different layers. The analysis highlights the significant role
The article details a novel system for high-throughput, zero-copy data movement between network interface cards and GPUs, achieving speeds of hundreds of gigabits per second. It explores the architecture and implementation strategies for enhancing data pipeline performance in high-performance comput
The article explains the decision to transition from a single AI agent to a multi-agent pipeline architecture, using text-to-SQL as a practical use case. It details the reasoning behind this shift and describes the construction of the multi-agent system.
The article presents OpenModeling Core, an open-source project centered on multidimensional modeling tailored for financial applications. It highlights the framework's capabilities in structuring complex financial data for analysis.
The article examines performance issues within Kafka share groups, focusing on pathological fetch waits. It explains how using share.acquire.mode=record_limit combined with fewer consumers than partitions and various forms of partition skew can lead to subpar performance. The post details the diagno
This article investigates the circumstances under which data significantly contributes to the engineering and improvement of automated AI agents. It likely discusses how data can be leveraged for agent training, evaluation, and overall system robustness.
The article describes an architectural approach for Retrieval-Augmented Generation (RAG) pipelines, focusing on anchor detection. It outlines a strategy involving parallel detectors for information retrieval, followed by a consolidated LLM call. The retrieval method prioritizes keywords, then table
This project, Autoops, describes a system for managing multi-region data infrastructure and service meshes, with its operations driven by a Makefile. It details the architecture and the pragmatic approach to orchestrating complex distributed systems.
This article explores the infrastructure required for accountability and governance under the EU AI Act. It delves into the technical considerations for building systems that ensure compliance and provide oversight for advanced AI models.
The article discusses the architectural requirements for AI agents, arguing for the necessity of a command-line interface in addition to the Model Context Protocol (MCP) server. It examines how MCP enables agents to interact with data systems but highlights a broader need for CLI-based interaction p
The article introduces Clai, an open-source project designed for context engineering, aiming to enhance the terminal experience for power users. It outlines how the system manages and utilizes contextual information within a command-line interface.
The article announces the general availability of Self-Managed OAuth for developers on Cloudflare, but primarily focuses on the technical process behind this rollout. It describes how Cloudflare executed a zero-downtime migration of its core OAuth engine to enable this new feature.
This paper proposes an extension to SPARQL by introducing a generative query construct called `GenOp`. This new operation allows SPARQL queries to invoke a language model and generate typed solution mappings, while preserving the fixed-dataset assumption for query semantics.
The paper introduces a Unified Dominance Graph method for Interval-Predicate Approximate Nearest Neighbor Search (ANNS). This technique addresses the need for hybrid queries in applications such as temporal databases, financial data analysis, and retrieval-augmented generation.
This paper considers an oracle that processes a limited batch of records to cluster entities referring to the same real-world object. It studies methods to interrogate such an oracle for resolving entities in datasets significantly larger than a single batch.
This paper describes how Presto was extended to be GPU-aware, focusing on critical challenges such as efficient data transfer from storage to GPU operators. It also addresses enabling data exchange between operators without leaving GPU memory, even in a distributed query environment.
This paper observes that time-series, geospatial, and ontology systems all maintain hierarchies, such as day <= month <= year, zip <= city, and is-a / part-of relationships, and typically index them separately. It proposes a unified index for these subsumption posets, focusing on order testing workl
This research investigates whether aggregate structural tests, specifically Laplacian interlacing for rejecting candidate subgraphs, can accelerate continuous subgraph matching. It builds on spectral filtering's success in pruning for static subgraph matching.
This paper examines query abstraction in an ontology-based data access (OBDA) setting, where multiple data sources are integrated through mappings to an ontology. It specifically considers an OBDA framework based on existential rules and the certain answer semantics.
This paper discusses the rapid evolution of memory systems for large language model (LLM) agents, which have expanded beyond simple retrieval-augmented mechanisms. These systems now support persistent information storage, retrieval, update, consolidation, and dynamic lifecycle governance.
This paper presents LGTD, a novel method for decomposing time series into trend, seasonal, and residual components without requiring a predefined season length. This technique is fundamental for applications such as anomaly detection, change-point analysis, and forecasting.
This paper introduces TSseek, a technique for regular expression-based similarity search in distributed time series datasets. It addresses the limitation of existing methods that typically require a precise sequence of values as the query input.
This paper presents ORQ, a system designed for collaborative analysis of large private datasets using cryptographically secure multi-party computation (MPC). ORQ offers strong protection against semi-honest or malicious parties and efficiently evaluates relational queries.
This paper investigates the computational complexity of evaluating Boolean query Directed Acyclic Graphs (DAGs) over text fields, which are increasingly employed by modern AI agents for neuro-symbolic reasoning workflows. It specifically focuses on the P-completeness of inverted index traversal.
This arXiv paper introduces ErrorLLM, a framework designed to improve the accuracy of text-to-SQL generation by addressing common errors. It focuses on the SQL refinement task, detailing how to model and correct erroneous SQL queries produced by large language models. The work aims to enhance the re
This arXiv paper presents a controlled study exploring the impact of knowledge-graph grounding on large language models, specifically in clinical question answering. It investigates whether structured knowledge benefits LLMs only for information outside their training data, following up on reports t
This article introduces BitVanes, a zero-trust, local-first ETL engine for RAG pipelines built with Rust, WASM, and Apache Arrow. It processes sensitive documents locally for parsing, PII scrubbing, chunking, and vectorization, addressing concerns about shipping raw data to cloud services.
The article explores a method for measuring search ranking quality by employing LLMs to judge Normalized Discounted Cumulative Gain (NDCG). It presents an approach to leverage large language models for evaluating the effectiveness of search algorithms.
This article argues that AI agents require real-time analytics in addition to vector search for effective operation. It proposes that Apache Doris addresses this need by providing a unified real-time engine capable of native hybrid search tailored for agent workloads.
The article announces the launch of the FFASR Leaderboard, designed to benchmark Automatic Speech Recognition (ASR) models against real-world audio data. It outlines the methodology for evaluating ASR performance in practical conditions, moving beyond controlled datasets. The leaderboard aims to pro
The article looks ahead to the upcoming Postgres 19 release, detailing the new features and improvements anticipated in this version. It covers various quality-of-life enhancements and potential advancements that day-to-day Postgres users will find beneficial. The discussion also touches upon areas
This article details how Visa implemented conversational BI agents using LibreChat and ClickHouse Cloud for Authorize.net payments data. The solution transformed multi-day reporting into sub-second queries, saving users 8-10 hours weekly.
This article explores methods to achieve state-of-the-art inference latencies, specifically focusing on the application of speculative decoding. It aims to provide insights into optimizing performance for machine learning models.
The article introduces the LLM-CTF benchmark dataset, comprising 2,639 real data points derived from NeurIPS research and new experimental runs. This benchmark aims to provide resources for evaluating LLMs in Capture The Flag scenarios.
This GitHub pull request introduces a draft proposal for integrating permissioned data capabilities into the ATProto, outlining design considerations and mechanisms for controlling data access.
The post investigates an 'expert-aware' quantization method designed to compress machine learning models to nearly Q2 size while retaining quality comparable to Q4 quantization. It discusses the technical approach and potential benefits for model deployment.
This article introduces `pg_graphwright`, a new Postgres extension designed to function as a knowledge graph index. A key feature is its ability to inherit row-level security from the underlying Postgres tables, simplifying access control for graph data.
This article outlines best practices for operationalizing data orchestration, specifically addressing considerations for DevOps teams, infrastructure setup, and managing code locations within an orchestration framework.
This architecture guide outlines approaches for real-time hyper-personalization systems. It details potential system designs for delivering highly customized experiences at scale, expected by 2026.
This guide explains how to eliminate training-serving skew in ML pipelines, a common issue degrading accuracy and increasing costs. It proposes a unified kappa architecture leveraging Apache Flink and Iceberg for real-time streaming ML.
The article explores combining the Origin Private File System (OPFS) with Pyodide to create a test harness for in-browser execution. It details the technical challenges and solutions for running Python environments directly within the browser, leveraging local storage capabilities for data processin
This article from WarpStream details their application of AI coding agents within their product development lifecycle. It explains how these agents are utilized effectively without compromising the system's uptime and reliability.
The article analyzes a major reliability incident at Coinbase where the global trading service lacked automated zone failover. It delves into the architectural implications of this failure and discusses the lessons learned regarding system resilience.
Meta details the engineering challenges involved in designing ultra-narrow batteries for AI-powered smart glasses, such as the Ray-Ban Meta. The article discusses how to provide sufficient energy to support features like cameras, speakers, displays, and AI workloads within the compact form factor of
The article proposes a mental model for enterprise Retrieval Augmented Generation (RAG) systems, suggesting that retrieval should be viewed as a filtering process rather than string search. It details strategies like filtering line_df and toc_df and expanding context from small anchors for more effe
This article, part of an Apache Iceberg Masterclass, explains what lakehouse catalogs are, their importance, and how to select among various options. It builds on previous discussions of the write process and atomic commits facilitated by catalogs.
This article introduces CUGA, a lightweight harness designed for building agentic applications. It provides two dozen practical examples demonstrating how to construct and implement real-world AI agents.
This post introduces SQLBuild, an open-source tool designed to optimize existing dbt projects. It functions by identifying and building only models that have changed, thereby skipping unnecessary rebuilds and reusing existing production tables to save compute resources. The project is fully open sou
The article addresses significant CPU costs incurred by Protobuf parsing overhead in data pipelines handling large volumes of structured data. It proposes an alternative zero-copy backend for Protobuf to mitigate deserialization bottlenecks, particularly when only partial data access or passthrough
This arXiv paper introduces ReSequel, a method for LLM-assisted query rewriting and optimization. It explores using templatization and sampling to transform SQL queries into semantically equivalent, more efficient forms, building on heuristic and cost-based optimization.
This arXiv paper introduces RAIDS, a framework for rethinking data systems as responsible intelligent infrastructure. It addresses the gap in responsibility mechanisms as data systems evolve into decision-making tools, discussing the need for sufficient support, satisfied constraints, and actionable
This arXiv paper addresses the absence of a principled I/O cost model for disk-resident learned indexes. It proposes a cache-aware I/O cost model, which is essential for effective index tuning and query optimization in database management systems.
This arXiv paper investigates when columnar scans, which involve decompression, filtering, and aggregation, are limited by memory bandwidth rather than compute. It presents a predictive decode-throughput law and validates it across different hardware configurations to identify performance bottleneck
This arXiv paper examines disk-based interval indexes, which are crucial for managing lifespan or validity intervals in temporal databases. It proposes that various interval indexes can be unified by a fundamental corner structure, especially under the assumption of increasing ending times.
This arXiv paper explores enhancing Large Language Models with graph structures to improve their spatial search and reasoning abilities. It builds upon Retrieval Augmented Generation (RAG) to overcome current LLM limitations in complex, domain-specific spatial tasks.
This arXiv paper introduces SemCEB, a new benchmark for evaluating cardinality estimation within semantic operators that utilize multi-modal large language models. It focuses on SQL operators, like filters and joins, where predicates are defined by natural language instructions, which is crucial for
This arXiv paper details a set-theoretic approach designed to identify logic bugs in the inner join optimizations performed by database management system query optimizers. It focuses on the crucial role of join optimization in determining efficient query execution strategies.
This arXiv paper proposes a new compositional language for property graphs. It aims to address the lack of compositionality in standardized graph query languages such as GQL and SQL/PGQ, which is a significant limitation when querying knowledge graphs. The paper presents both theoretical aspects and
This arXiv paper introduces SQLConductor, a framework that employs search-to-policy learning for step-wise Text-to-SQL orchestration. It aims to improve natural language access to relational databases, particularly in complex real-world settings where coordinated reasoning is essential. The approach
This arXiv paper proposes a method for testing Large Language Models (LLMs) by using counterfactual relational data. The research investigates whether LLMs answer natural-language questions over structured data by interpreting the provided table or by recalling previously learned real-world facts. T
This arXiv paper presents the Generative Responsible AI Data Evaluation Schema (GRAIDES), designed for AI assurance in local government. The schema aims to establish well-governed, measurable evidence of generative AI performance and safety. It addresses challenges related to fragmented and inconsis
This arXiv paper presents FireDataForge, a unified framework for retrieving and integrating multi-source wildfire geospatial data. The framework aims to overcome the significant preprocessing burden associated with diverse data formats, coordinate systems, spatial resolutions, and temporal cadences,
This arXiv paper explores Relational Deep Learning (RDL) models that represent multi-tabular databases as temporal heterogeneous graphs for end-to-end representation learning. It identifies significant generalization obstacles in current RDL approaches and proposes universal encoders to create modul
This arXiv paper presents TACO, a system that leverages Large Language Models to generate accurate and informative column descriptions for tabular data. Such descriptions are vital for various downstream Natural Language Processing tasks, including Natural Language to SQL, table question answering,
The article presents Netflix's initial research into achieving more controllable artificial intelligence for video editing workflows. It explores methods and challenges in enabling greater user direction over AI-powered creative processes.
This article details the weekly release process for huggingface_hub, emphasizing the integration of AI-driven automation and open-source tools. It highlights the crucial role of human oversight in maintaining quality and consistency in a rapid development cycle.
Redpanda SQL introduces bridge queries, a feature that enables querying both live streaming topics and historical Iceberg tables concurrently. This approach aims to eliminate the typical compaction overhead associated with integrating fresh and historical data.
This article, titled 'CostBench', presents a comparison of Snowflake and ClickHouse Cloud for real-time analytics. It evaluates both platforms across the entire data path, including continuous ingest, query-ready data maintenance, freshness, query latency, and total cost.
The article details updates in the latest pg_clickhouse releases, including support for Postgres 19, TLS, and regex functionalities. It highlights JSONB, date/time, and array function pushdown, along with HTTP result set streaming for reduced memory consumption.
The post analyzes prompt injection attacks through the lens of 'role confusion' in large language models. It examines how adversarial prompts manipulate an LLM's perceived identity or function, leading to unintended behavior, and discusses methods to mitigate this vulnerability in agentic systems.
This article details the process of porting the Moebius 0.2B image inpainting model to run directly within a web browser. It explores the technical steps involved in optimizing the model for client-side execution and how an AI assistant like Claude Code aided in the conversion process.
The article explains how Netflix utilized Kueue to streamline its batch compute infrastructure. It covers the architectural considerations and operational patterns employed to simplify the management of large-scale batch workloads.
Snowflake's ML team utilized Snowflake Postgres for their Online Feature Store, achieving 2.5 times lower latency and 7 times higher queries per second compared to Databricks Lakebase in production benchmarks. The article details this performance comparison for powering ML feature serving.
Cloudflare uncovered a bug in the open-source hyper HTTP library across multiple major versions. This discovery happened while rearchitecting their Images binding. The article explains how the bug was found and its implications.
This article argues that AI agents fail in production due to a lack of structure rather than insufficient autonomy. It implies a need for robust frameworks around agent operations to ensure trust, particularly concerning sensitive resources like database credentials.
Meta details its multi-year effort to adopt the AV1 codec for real-time communication at scale. The article covers technical and operational challenges encountered during deployment, including codec selection, device eligibility, rate control, and error resilience, and how these were addressed.
The article introduces 'semantic debt' as a situation where different teams derive conflicting numbers for the same metric. It argues that the rise of AI will force organizations to address this inconsistency more urgently.
The article introduces a strategy for enhancing enterprise RAG systems' ability to handle ambiguous user questions. It advises designing RAG agents to ask a single focused clarifying question, learn from the user's response, and subsequently infer defaults for similar future queries.
This article proposes a taxonomy for categorizing modern data storage and workload architectures, addressing the increasing convergence of transactional, analytical, and hybrid systems. It details how systems, workloads, storage tiers, data visibility, and durable copies interact within these evolvi
This article explores the use of local machine learning models for triaging issues within the OpenClaw repository. It describes the implementation and effectiveness of using these models to automate parts of the repository management process, highlighting potential cost efficiency.
Spyne migrated its Change Data Capture pipeline from a self-managed Debezium and Kafka stack to ClickPipes and ClickHouse Cloud. This transition reduced table onboarding time from over 1.5 hours to minutes and successfully eliminated schema drift issues.
This article provides a detailed technical examination of sandbox startup latency, distinguishing between when a system has started and when it is truly ready for use. It delves into the underlying factors contributing to delays in application readiness. The content offers insights into performance
This release candidate for `sqlite-utils` introduces robust support for database migrations, enabling programmatic management of schema changes. It also adds nested transaction capabilities, allowing for more granular control over complex data operations and error handling within SQLite databases.
This announces the release candidate for `sqlite-utils` version 4.0, a Python CLI and library for SQLite. Key new features include robust database migration capabilities and support for nested transactions, enhancing complex data management and schema evolution.
The article discusses a method for provisioning temporary Cloudflare accounts, designed to provide isolated environments for AI agents. It explores the architectural patterns and security considerations for enabling agents to interact with external services while minimizing risks and managing access
The article explains the concept of tool calling in AI agents, detailing how Large Language Models (LLMs) determine subsequent actions to interact with the external environment, whether by retrieving data or executing operations.
The article addresses the challenge of processing PDFs with visible but unstructured tables of contents for RAG systems, detailing two methods for reconstructing document structure and emphasizing a crucial, often-missed page-alignment step to enable section-level scoping.
This article outlines effective patterns for constructing cybersecurity evaluations, focusing on key components such as a sandboxed target environment, inputs designed to modulate task difficulty, integrated tool usage, and a robust grading mechanism.
This article examines seven significant obstacles preventing data teams from implementing self-healing data architectures. It discusses how AI technologies can be leveraged to overcome these barriers and make such autonomous data systems a reality.
This article describes techniques for making images within PDF documents searchable for RAG applications while minimizing processing expenses. It outlines an approach where image locations are identified, and only relevant images are converted into searchable text to control costs.
The article discusses the advanced development of VMAF (Video Multi-method Assessment Fusion) at Netflix, explaining why its initial version was deemed insufficient for evolving quality standards. It details the technical challenges and improvements made to enhance video quality assessment.
This article details the concept and implementation of speculative decoding, a technique used to accelerate large language model inference. It explains how a smaller, faster model can generate draft tokens that a larger, more accurate model then verifies, significantly reducing latency and compute r
This article examines the classic thundering herd problem as it manifests in agentic AI systems, highlighting how traditional mitigation strategies are insufficient. It discusses the unique characteristics of AI agent behavior that exacerbate this issue and proposes new considerations for designing
This article introduces 'Datasette Apps,' a feature that allows users to embed and host custom HTML applications directly within a Datasette instance. It details how this functionality enables richer data exploration interfaces and custom dashboards alongside the core data publishing capabilities.
This announcement details the 0.6a0 release of `datasette-acl`, a plugin providing Access Control List capabilities for Datasette. It covers new features for fine-grained permission management and policy enforcement, enhancing data security and compliance for published datasets.
This article introduces MosaicLeaks, a framework designed to test the security and privacy capabilities of research agents. It investigates the potential for AI agents to inadvertently reveal sensitive information and discusses strategies for ensuring data confidentiality within agentic systems.
Cloudflare details the technical architecture of its multi-stage vulnerability discovery harness and automated triage loop. The post covers state control management, adversarial review to reduce false positives, and methods for routing around LLM context limits.
This case study details CodeRabbit's implementation of LanceDB to power its AI-driven code review system. It describes how leveraging LanceDB for context engineering enhances the quality of code reviews.
This article explores methods for deploying high-performance distributed inference systems for large language models. It focuses on leveraging Ray Serve to manage and scale LLM inference across multiple computing resources efficiently.
This article explains the concept of late materialization within Lance Spark for handling large binary data. It details how this approach maintains lightweight references throughout query plans, only materializing bytes during the write phase.
This article investigates fine-tuning techniques for large language models beyond the popular LoRA method. It evaluates alternative approaches and their potential to surpass LoRA's performance and efficiency in various fine-tuning scenarios.
This article discusses methodologies for benchmarking the agentic capabilities of open large language models using custom evaluation tooling. It explores criteria for determining whether a model exhibits sufficient agency for specific tasks and provides guidance on developing relevant benchmarks.
This article details how Redpanda's Cloud Topics implement adaptive write request scheduling using the buddy allocator algorithm. The system balances batching efficiency with latency and cost considerations to optimize performance.
The article discusses strategies for importing and vectorizing data at scale with Weaviate. It covers server-side batching, retries, the blobHash data type, and multimodal ingestion, explaining when and how to use each with code examples.
METRO Markets replaced a failing Hadoop-based data stack by building a unified data platform on ClickHouse Cloud. This new platform now supports various functions including data warehousing, real-time seller analytics, credit risk modeling, observability, and company-wide AI initiatives.
Appcues migrated its real-time segmentation platform from Snowflake and Airflow to ClickHouse Cloud to manage 1.31 PB of data. This migration resulted in a 90% reduction in P95 query times, a 99% decrease in ingestion latency, and a 23% cut in overall analytics spending.
This article presents an evaluation of GLM-5.2, asserting its position as a leading text-only open-weights large language model. It likely includes performance benchmarks, architectural highlights, and a comparison against other prominent open-source LLMs in various text-based tasks.
The article describes payment fraud detection as a significant data-intensive challenge within financial services. It explores methods employed by banks and businesses to prevent fraudulent transactions.
Enterprise agentic analytics involves enabling AI agents to perform multi-step analysis across diverse enterprise data sources, including data warehouses, databases, object stores, and SaaS tools, each with distinct access rules.
Cloudflare's Agents SDK is now a runtime any agent framework can build on. The company is opening up the Agents SDK primitives, with Flue introduced as a first framework targeting the SDK. Agents are also rolling out in the dashboard.
Snowflake supports the Agentic Resource Discovery (ARD) specification, an open protocol developed with Microsoft. This protocol aims to standardize how AI agents are cataloged, searched, and discovered across enterprises.
The Cloudflare One stack is a library of agent skills designed to equip AI agents with the knowledge needed for planning, deploying, and managing a Zero Trust environment. This approach eliminates the need for migration calls.
This article introduces a new LanceDB-backed memory plugin that provides durable, semantic recall across sessions for the Hermes Agent. It includes benchmarks and a hands-on walkthrough demonstrating remember, recall, and forget functionalities.
This article presents a Rust-based benchmark comparing the metadata performance of Lance, Delta Lake, and Apache Iceberg when used with S3 and S3 Express storage. It explains why Lance is optimized for object storage metadata operations.
The post details deploying ML models from the Hugging Face Hub onto robot hardware. It discusses using Strands Agents and the LeRobot framework to bridge models to real-world robotic applications.
This article details how writing transaction commit logic as a SQL view can achieve higher throughput compared to control-plane approaches. It explains how incremental view maintenance enables fast resolution, suitable for interactive timescales around 30ms.
The article announces the release of DuckDB 1.4.5 LTS (Andium), differentiating it from the concurrent 1.5.4 (Variegata) release. It provides an overview of the key updates and fixes introduced in this long-term support version.
This article announces the release of DuckDB 1.5.4 (Variegata), the latest non-LTS stable version. It highlights important updates and features distinguishing it from the 1.4.5 LTS (Andium) release.
This article discusses agentic resource discovery, enabling AI agents to autonomously search for and utilize relevant information. It covers how agents can expand their knowledge base and tool use capabilities through intelligent search.
This article presents a detailed post-mortem analysis and tactical guide on how a digital photo-frame company scaled its Postgres deployment to 226,000 transactions per second. It describes the issues encountered during a holiday peak and the specific strategies implemented to resolve them.
Open Electricity migrated approximately 1 billion rows of Australian energy data from Postgres and TimescaleDB to ClickHouse, achieving 92x faster queries on a fraction of the hardware.
The article presents a dbt migration guide, cautioning against merely replicating legacy patterns in new tools. It instead advocates for a 'start fresh' approach to ensure migrations deliver better outcomes.
The article explores the role of dbt's transformation layer in establishing trustworthiness for agentic data pipelines. It posits that this layer defines data correctness within autonomous data systems, even when AI agents manage pipeline execution.
The article positions context engineering as an essential skill for analytics engineers, explaining how existing dbt projects can be structured to provide vital context for AI systems. It offers a practical guide for dbt users.
This article presents a practical dbt guide for constructing an agentic data stack in the AI era. It outlines methods for preparing dbt projects to reliably support AI agents and prevent system instability, even as AI accelerates infrastructure development.
The article discusses the 'trust-speed paradox' in data teams, noting that while many use AI for code generation, few adequately verify its output. It aims to provide strategies for bridging this gap and improving governance in AI-accelerated data workflows.
Snowflake Postgres now includes data mirroring and pg_lake integration, enabling a native, pipeline-free method to synchronize OLTP and analytical data in near real time. This unifies applications, analytics, and AI capabilities.
Google DeepMind outlines an AI Control Roadmap for securing internal systems that utilize AI agents. This strategy combines established safeguards with real-time monitoring techniques to enhance the security posture of AI deployments.
This article, Part 6 of an Apache Iceberg Masterclass, details the precise steps an engine takes when writing data to an Iceberg table. It covers when a write becomes visible and how concurrent writers are managed, following a discussion on hidden partitioning.
This article proposes the Agentic Lakehouse as an architecture designed specifically for AI agents, differing from traditional lakehouses optimized for human analysts and predictable SQL. It identifies how current lakehouse designs, tuned by DBAs for known query patterns, are insufficient for AI-dri
The article explores four practical applications of Figma's MCP server, demonstrating its expanded role across the platform. It details how the server supports processes from updating dynamic content to facilitating design shipments to production.
This article argues that data processing is increasingly becoming a GPU-centric workload. It examines the underlying trends and technological advancements that are driving this transition in how data is processed.
OpenAI describes Deployment Simulation, a method designed to predict AI model behavior before release. This technique uses real conversation data to enhance safety assessments and improve the accuracy of model evaluations in a pre-production environment.
This article details architectural patterns for building audit-ready, EU AI Act-compliant agents with stateful stream processing on Apache Kafka and Flink. It outlines 7 states, 4 patterns, and a phased rollout strategy.
The article describes how LanceDB utilizes the flexible data evolution features of the Lance format to facilitate scalable feature engineering for multimodal datasets.
The post benchmarks OpenSearch and LanceDB for vector search, using COCO 2017 images with SigLIP embeddings. It measures ingestion throughput, query cost, storage layout, and overall infrastructure requirements.
The article guides readers through the operational mechanisms of AI inference and explains the foundational reasons for the development of optimization techniques within this field.
The article presents an Orchestration Maturity Model, explaining the evolution of data orchestration systems from focusing on jobs to managing data assets. It describes why enterprises are adopting systems like Dagster to transition to asset-aware data platforms that provide insights into data rathe
The author shares their experience with running local models, noting that performance has significantly improved. They detail testing various models such as Mistral 7B, Gemma 3, OpenAI OSS-20B, and Qwen 3 MoE on an M2 Mac with 64 GB RAM.
Neon has launched `neon.ts`, an infrastructure-as-code file designed for managing Neon projects. This tool enables users to declare Neon services, access type-safe environment variables, and configure branch settings, facilitating the provisioning of backend primitives for applications and agents.
This architectural guide explores when to use Apache Kafka and when to avoid its implementation. It discusses the core differences between event streaming and task queues to design scalable systems reliably.
Cloudflare's Security Insights system now processes over 120 scans per second, achieving a 10x increase in global scanning capacity. This was accomplished by optimizing Kafka consumers, Postgres queries, and their API without adding hardware.
Neon addresses `psql` availability issues across various operating systems and environments by reimplementing the `psql` client entirely in TypeScript. This new client is embedded directly within the `neonctl` command-line interface, ensuring its functionality even when the native `psql` is not inst
This article details achieving up to 67% cost savings in LLM inference by implementing prefill-decode disaggregation. It describes how this technique is applied using Ray and vLLM on AMD MI325X hardware to optimize computational resources.
This article provides an in-depth look at Fully Sharded Data Parallel (FSDP) for scaling machine learning model training. It explains how FSDP is implemented using PyTorch and Ray to efficiently distribute and manage model parameters across multiple devices.
The article describes the re-architecture of Figma's layers panel, implementing new computation and caching strategies. These changes resulted in a 30–50% improvement in interaction speed for large and complex files.
The article explores a recent trend among engineering departments to reduce spending associated with AI initiatives. It examines the potential reasons behind this shift and its implications for technical teams.
This article details major deployment strategies used in production, including their operational mechanisms, associated costs, and appropriate use cases. It covers methods ranging from big-bang to progressive delivery techniques.
The article compares Text-to-SQL and Agentic Analytics, noting that leading large language models achieve 60-70% accuracy on complex SQL queries according to the BIRD benchmark. It discusses accuracy differences between simple and multi-join queries and examines the architectural requirements for ev
This article introduces Agent Skillsan as a method for debugging Ray applications on Anyscale without extensive log analysis. It explores how this agent-based approach streamlines the identification and resolution of issues in distributed systems.
This post explores methods to accelerate FlashAttention-4 specifically for inference workloads in large language models. It details technical optimizations and algorithmic adjustments aimed at reducing computation time and memory footprint during the attention mechanism calculation, leading to impro
The article presents a benchmark comparing Apache Doris 4.1's ASOF JOIN performance against ClickHouse and DuckDB. Apache Doris 4.1 demonstrates superior performance across all eleven tested scenarios for this specific join type.
This is the second part of a series on profiling PyTorch models, focusing on optimizing performance from nn.Linear layers to fused MLPs. It details techniques for identifying and reducing bottlenecks in neural network execution.
This article explains the architecture of Redpanda's metastore for Cloud Topics, detailing how it enables features such as offset lookups, complete cluster restores, and cross-region read replicas. It positions the metastore as a foundational component for future system capabilities.
Stripe Projects is expanding with new integrations, additional providers, and customizable developer controls to address challenges agents face beyond code writing. This initiative aims to enhance agents' ability to independently write code and integrate with APIs like Stripe's.
The article presents case studies from Adyen, Xoople, Criteo, and BMW detailing their approaches to scaling AI workloads using Ray. It highlights Adyen's experience training a Transaction Foundation Model on 51 trillion tokens. The discussion covers practical aspects of deploying and managing large-
The article explores architectural patterns for integrating a database, specifically CockroachDB, to manage the memory, context, and control aspects of autonomous AI agents. It details how the database can serve as a persistent store for agent state, conversational history, and operational parameter
Google DeepMind introduces DiffusionGemma, a new model designed for text generation. The article claims this model achieves a 4x speed improvement compared to previous methods, offering significant advancements in text generation efficiency.
This article, part of a series on Kafka consumption, explores client-local parallelism as a method for scaling. It differentiates between broker-visible and client-local approaches, detailing how execution mechanisms such as threads, virtual threads, or async tasks enable local parallelism.
This article from Spotify Engineering outlines the development of a "Context Layer" to encode domain expertise, which powers their internal Data Assistant. It discusses how this system helps address complex data problems by providing contextual understanding and streamlining data access for users.
This article clarifies the distinction between a semantic layer and a data catalog, two terms often used interchangeably despite serving different purposes. It explains their unique roles in a data architecture and how each handles metadata to improve data understanding for both human and machine co
This article announces the release of PostgreSQL 19 Beta 1, highlighting key new features in this major version. These include graph query support, enhancements for faster data inserts, the introduction of pg_plan_advice, and capabilities for parallel-worker autovacuuming and online toggling of data
The article features how companies like Torc, Discord, Cubist, and Coinbase manage and scale their AI systems using Ray. It specifically highlights Torc's achievement of 90% GPU utilization for their ML workloads. The content explores real-world strategies for optimizing resource use in AI deploymen
This article focuses on strategies for managing the operational costs associated with deploying and scaling agentic AI systems. It discusses factors contributing to cost blowouts, such as token usage multipliers and context window management, offering practical approaches to optimize spending in pro
This article details how Airbnb evolved its data architecture to support multiple distinct product lines, moving beyond a monolithic approach. It describes the architectural changes, challenges encountered, and solutions implemented to scale data infrastructure effectively.
This article presents insights from John Kucera, Salesforce's CPO of Agentforce, on distinguishing successful enterprise agent deployments from those that fail to deliver sustained business value. It draws from the experience of 20,000 agent deployments within Salesforce.
Google DeepMind presents Gemma 4 12B, a new unified multimodal model. A key architectural detail highlighted is its encoder-free design, representing a novel approach in multimodal AI systems.
This article, part of an Apache Iceberg Masterclass, details the hidden partitioning feature within Iceberg. It explains how this capability eliminates the need for users to understand physical data organization and prevents costly accidental full table scans during queries.
This article addresses the governance gap arising from AI agents executing hundreds of queries per minute without human review. It discusses how traditional access controls are insufficient and explains how a semantic layer can provide the necessary control for what AI agents can access.
This article demonstrates an AI agent constructing a 3D gallery by chaining together two distinct Hugging Face Spaces. It illustrates the agent's ability to orchestrate multiple tools to achieve a complex creative task.
Cloudflare details the architecture behind Project Glasswing, emphasizing its importance in defending against vulnerabilities over rapid patching. The article describes the specific threats this architecture addresses and how Cloudflare implements it as its internal "customer zero."
The article describes Discord's project to migrate its voice and video services onto Cloudflare's edge network. It covers the technical process of achieving closer servers and reduced ping times across regions, along with specific bugs encountered during the implementation.
This article outlines four essential pillars for effective AI agent governance at scale: identity, authorization, observability, and accountability. It emphasizes the necessity of robust governance infrastructure beyond just improving agent models for enterprise deployment.
The article explores why AI proof-of-concepts often fail in production, attributing the gap to issues within the underlying data model rather than the machine learning model itself. It suggests that a robust data model is crucial for successful production AI deployments.
This article explores strategies for managing and optimizing LLM token spend in production environments, specifically focusing on smarter routing techniques. It includes insights from the co-founders of Kilo, an open-source coding agent that frequently encounters these challenges.
The article describes a new human-augmenting agentic workflow developed at Netflix to enhance causal inference capabilities. It details the architecture and operational patterns that allow AI agents to collaborate with humans on complex analytical problems.
The article details the architecture of Netflix's personalized notification system, which leverages the 'Thinking Fast & Slow' framework. It describes how both immediate and deliberative processing contribute to tailoring notifications for individual users.
Apache Doris 4.1 implements a per-user identity mode, forwarding individual user identities to Apache Polaris, an Iceberg REST Catalog. This design eliminates the need to route all queries through a single shared service account, enhancing security.
This article outlines the development of Sitar-agent, a dynamic configuration sidecar designed for high reliability and scalability at Airbnb. It covers the architecture, challenges, and solutions involved in building such a critical infrastructure component.
Presented as a side-quest in a series about Kafka share groups and parallel consumption, this article focuses on the fundamental differences between broker-visible and client-local parallelism. It examines how various configurations and behaviors specifically influence parallel consumption within sh
This article announces the release of Multigres v0.1 alpha to the open source community. Multigres aims to provide Vitess-grade horizontal scaling, high availability, and operational simplicity for Postgres.
The article investigates the typical points of failure and production incidents that arise when deploying agentic AI systems in real-world environments. It outlines common hurdles faced by enterprise AI teams beyond initial impressive prototypes, detailing the complexities of moving agents from deve
Meta is introducing Instantaneous PowerLoss Storm, a new testing paradigm within its infrastructure for handling and mitigating instant or zero-notice power loss in data centers. The article shares how Meta built readiness to tolerate instant failures into existing systems with defense-in-depth stra
Spotify's chief architect discussed strategies for enhancing the effectiveness of both engineering teams and AI agents. The presentation covered methods for scaling developer experience within the organization.
This article explores OpenAI's approach to building its data agent, emphasizing that the primary challenge in data analysis lies in discovering relevant tables and understanding their semantic usage, rather than SQL authoring. It delves into how they tackle these complex data discovery and interpret
The article argues that APIs are critical for AI agents, which interact more effectively with programmatic interfaces than graphical dashboards. It emphasizes that platform functionality visible in user consoles should also be accessible via open API endpoints. This approach supports agentic workflo
This article details a benchmark study on optimizing training data reads, achieving a 20x speedup by integrating Alluxio with Ray Data. It focuses on performance improvements for data access patterns, particularly in cross-region scenarios. The content covers the technical aspects of reducing data r
The article describes Netflix's approach to dynamic repartitioning specifically tailored for time series data workloads. It details the mechanisms and architectural considerations for optimizing data layout and query performance on time-ordered datasets.
The article explores a hybrid modeling strategy within Apache Doris for managing dynamic, schema-evolving agent observability logs. It leverages the VARIANT data type and native inverted indexes to achieve high performance for this type of data.
Postgres 19 will introduce support for SQL/PGQ, enabling users to declare property graphs over existing tables. This allows for pattern matching with Cypher-like syntax within Postgres.
Stripe explains its rapid implementation to enable businesses to leverage the Visa Digital Commerce Authentication Program (DCAP). The initiative focused on helping businesses achieve interchange savings while maintaining strong authorization rates.
The article describes how LanceDB processes raw multimodal data to create reproducible, training-ready datasets. It covers features such as search, filtering, deduplication, sampling, and versioned curation workflows.
This article presents a novel problem-solving methodology, drawing insights from geographical concepts to address limitations encountered with traditional historical data approaches. It discusses how applying these alternative frameworks led to effective solutions for complex engineering problems.
OmniNode's founder discusses the development of their AI agent workflows, detailing how Redpanda is utilized for scaling these operations. The article specifically highlights how data contracts are employed to manage and prevent topic name drift within their streaming infrastructure.
This article describes how Brooklyn Data implements Compass to facilitate self-service analytics within Slack, aiming to decrease response times for data queries, enhance data discoverability, and establish governed access to operational and financial data across the organization.
The article argues that scaling reinforcement learning applications in production primarily presents an infrastructure challenge, not just an algorithmic one. It explores the system design considerations and engineering requirements needed to effectively train and deploy RL agents at scale, covering
NVIDIA Vera is launching with Redpanda as part of the ecosystem, aiming to deliver 5.5x lower latencies for agents running in mission-critical environments.
Streambed is a tool that streams Postgres data to Iceberg on S3 and supports the Postgres wire protocol.
The author presents a demo of rendering 1 billion rows (14 billion cells) in the browser using DuckDB and Glide Data Grid.
This article discusses the limitations of embeddings in RAG systems, noting that while they handle synonyms and paraphrasing well, they can fail on negations, exact identifiers, and company-specific acronyms. It suggests alternative approaches for when these failures occur.
The article presents a cost control layer for RAG systems that combines semantic caching, query routing, token budgeting, and circuit breaking. The approach reportedly achieves an 85% reduction in LLM costs without significantly impacting answer quality.
A 10 TB events table partitioned by month was the right call two years ago. Now your data volume has grown tenfold, your team runs daily SLA dashboards, and every query that touches "last 7 days" is scanning an entire month's worth of files. In a traditional Hive-style warehouse, fixing this means a
The article provides a complete walkthrough for constructing an autonomous vehicle perception model training pipeline. It demonstrates building this pipeline from raw data to a trained model using LanceDB and the Multimodal Lakehouse.
The article presents a hands-on walkthrough on integrating LanceDB and DuckDB to facilitate multimodal data querying using SQL. It covers joining data across multiple tables and materializing results back into LanceDB.
This blog post demonstrates the new features available in DuckDB v1.5.3 for the DuckDB-Iceberg extension, even while the team focuses on DuckLake v1.0 and Quack.
The article discusses the infrastructure required to build a fast and reliable scientific agent using local open-weight models, vLLM, and long-context infrastructure.
Here’s how we built Town Lake, Cloudflare's unified analytics platform, alongside Skipper, an internal AI agent running on top of it.
The traditional data lakehouse was designed for human analysts. Every architectural decision, from how performance is tuned to how business context is stored, assumed that a person would be sitting at the end of the pipeline, writing queries, interpreting results, and carrying those results into dec
Catalog governance is the biggest bottleneck in building a multi-engine lakehouse. When you query the same Apache Iceberg tables with Spark, Flink, and Dremio, synchronizing permissions and access credentials across different engines is traditionally a manual, error-prone chore. Apache Polaris solve
Apache Doris 4.1 integrates native Approximate Nearest Neighbor (ANN) vector indexes, including IVF and IVF_ON_DISK, directly into its OLAP engine. This integration achieves 900 queries per second at 97% recall when benchmarked using VectorDBBench.
In 2025, solo founders in the top decile generated 61 times the revenue of the median solo founder in their first six months. We analyzed the data to understand what drives that gap.
Choosing the right concept is only half the job. Plenty of teams have adopted the lakehouse model, picked open formats, and still built systems that fail when AI agents start querying them at scale. The Agentic Lakehouse architecture solves a specific problem: how do you structure a data platform so
The post explains how to set an appropriate value for max.poll.records when using Kafka share groups, revealing the relationship between group.share.partition.max.record.locks and the number of consumers per partition.
This article describes a systematic approach to leveraging LLMs for securing source code. The process involves constructing a threat model, identifying potential vulnerabilities, verifying findings, triaging discovered issues, and applying necessary patches.
Redpanda SQL is generally available, allowing direct SQL queries against live Kafka topics and Iceberg history without requiring a separate ETL pipeline.
A fintech company automated its compliance reporting using agentic AI to extract data from multiple sources and synthesize reports, reducing weekly reporting time from 2 days to 2 hours.
WarpStream details their approach to safe shipping across 24 control plane regions, discussing deploy trains, shadow services, zonal Kubernetes clusters, and async reconciliation loops.
A mid-tier SaaS provider automates cloud support triage using a 5-agent workflow, improving ticket validation, routing, and SLA compliance.
We’re introducing SilverTorch, a reimagining of recommendation systems that unifies all retrieval components for user generated content under a unified architecture. SilverTorch shows up to 23.7x higher throughput compared to the state-of-the-art approaches. It’s also showing 20.9x more compute cos
This is Part 3 of a 15-part Apache Iceberg Masterclass. Part 2 covered the metadata structures of all five table formats. This article focuses on exactly how query engines use Iceberg's metadata to avoid reading data they don't need. The single biggest performance advantage of Iceberg over raw data
The post examines the impact of tuning max.poll.records in Kafka share groups, using Kafka 4.2.0 and Dimster for testing.
The article contrasts Retrieval-Augmented Generation (RAG) and agents as solutions for accessing company data with LLMs, highlighting their distinct problem-solving approaches.
Apache Iceberg is not a static format. The spec version number stamped into every table's metadata controls which features that table can use, which engines can read it, and how efficiently row-level changes are handled. The jump from Apache Iceberg V2 to V3 introduces deletion.
Models can lose accuracy after retraining, and reproducing the exact training dataset from months ago can be difficult due to data lake changes. Apache Iceberg solves this by providing data versioning capabilities, allowing you to track and reproduce specific datasets used for training.
The post presents a benchmark comparing the overhead of Kafka Consumer Groups and Share Groups using Dimster, a performance benchmarking tool for Kafka.
This article describes Materialize's success since February 2026 in using LLM-based coding agents, primarily Anthropic’s Opus 4.6 and 4.7, to find bugs in existing code and open pull requests. It will cover system considerations for implementing such an approach.
This post describes how Pinterest engineers optimized their systems to handle user-sequence data more cost-efficiently, faster, and with improved usability. It likely covers specific architectural changes and engineering techniques implemented to achieve these gains.
This article explores a re-imagined approach to ML operations through the lens of "Agent Skills," proposing a new maturity model for on-call responsibilities. It discusses how AI agents can potentially transform MLOps practices. The content delves into conceptual frameworks for improving operational
DuckDB users can now query Lance datasets using SQL through the CLI or SDKs, enabling AI and retrieval workload capabilities; this post highlights Lance as a good option for vector storage and querying.
Use Weaviate's built-in MCP server to give Claude Code, Cursor, and VS Code hybrid search over your codebase and docs. No glue code.
SID-1 is an agentic search model that is 24x faster than GPT-5.1-high, 374x cheaper than Sonnet 4.5, and achieves 1.9x higher recall than traditional RAG pipelines. The article explains how it was trained using large-scale RL on turbopuffer.
DuckDB v1.5.3, while a patch release, includes several important new features; the complete release notes are available on GitHub, with installation instructions provided.
This article describes how Airbnb built and scaled a unified knowledge graph infrastructure to manage its complex identity graph. It covers the architectural patterns, challenges, and solutions for entity resolution and connecting disparate identity information at scale.
The post details how Redpanda Cloud Topics manages the lifecycle of temporary L0 objects and safely deletes them without data loss or excessive storage costs.
Docusign reports reducing dbt unit test authoring time from 5 hours to 30 minutes using a structured AI-assisted framework.
How ChatFeatured cut analytics query times from 2.5 minutes to under a second by migrating from PlanetScale Postgres to Postgres managed by ClickHouse — in just 30 minutes.
The paper proposes a novel ontology for relational database data lineage to address the challenges of modeling lineage, especially with incomplete or missing dependencies between database objects.
The paper explores the potential of language models and graph neural networks to serve as foundation models for relational databases, aiming to enhance deep learning applications on relational data.
The paper presents a gradient-based approach for join ordering, which is a computationally complex problem that critically impacts query execution performance in databases.
The article announces that end-to-end encryption is now enforced for all voice and video calls on Discord, effective March 2026. Discord's VP of Engineering discusses the importance of this multi-year commitment.
Digital sovereignty in streaming demands architectural guarantees, not policy promises. The post discusses how BYOC, schema controls, and open protocols satisfy global regulations.
The D. E. Shaw group replaced its previous observability platform with ClickHouse to handle high-cardinality metrics at scale, achieving 7x better query performance and enabling multi-year capacity planning across millions of compute workloads.
This guide covers three critical decisions for production RAG systems: chunk shaping, embedding selection, and ANN index scaling, bridging the gap between demo retrieval and real-scale deployments.
Learn how to design governance directly into event streaming systems. The post covers schemas, lineage, security, retention, and compliance patterns for reliable data platforms.
A deep dive into how pg_clickhouse's Foreign Data Wrapper decides what SQL to push down to ClickHouse versus execute locally in Postgres .
The article argues that enterprise AI systems are entering a phase where inference design matters as much as model capability itself.
Why does high cardinality break Prometheus but not ClickHouse? In Part 1, we explore the architectural tradeoffs of Prometheus and other series-based systems, showing how cardinality impacts memory, ingestion, querying, and operational stability at scale.
The post presents a critical analysis of MRC's three counterintuitive design decisions behind OpenAI's 131,000-GPU training fabric, including the networking mathematics that make them work and their implications for the AI infrastructure community.
Cloudflare's billing pipeline slowed down due to lock contention in ClickHouse's query planner after a partitioning change. The post details how they identified and fixed the bottleneck.
This article explores the architectural challenges and solutions for designing data pipelines that can process multimodal datasets efficiently at scale. It delves into strategies for managing diverse data types and large volumes within machine learning workflows. The content covers system design pri
This article introduces Viaduct 1.0 and outlines Airbnb's vision for its data mesh architecture. It details the principles, components, and future direction of their decentralized data management approach, highlighting how Viaduct serves as a key enabler.
Learn how ClickStack’s new SQL-powered charting and alerting unlock anomaly detection, rolling baselines, and advanced observability workflows directly on top of ClickHouse, without relying on external tooling.
This article breaks down exactly how each format organizes its metadata, which determines how fast queries start planning and how efficiently concurrent writes occur.
The article explores Databricks' implementation of rate limiting at scale, focusing on shrinking the critical path and the necessary accuracy tradeoffs.
This article presents an engineer's guide from Pinterest on establishing a testing process to optimize AI agent performance and reliability. It likely details methodologies and best practices for evaluating and improving agent skills.
Meta's engineering teams revamped their data ingestion system to enhance reliability at scale, migrating from a legacy system to a new architecture.
Learn how ClickHouse Cloud's Join table engine enables fast, updatable in-memory lookups for dimensional modeling — with automatic upserts, deduplication, and data compaction powered by ReplacingMergeTree under the hood.
The article describes how Figma's engineering team evolved their data pipeline to handle growth, reducing latency from multiple days to real-time.
Cloudflare investigated a performance issue caused by CUBIC's congestion window getting stuck at its minimum, identifying the root cause as incorrect measurement of idle periods. The fix involved accurately distinguishing RTT wait times from application idleness.
The post discusses techniques to cut inference cold starts by 40x using LP, FUSE, C/R, and cuda-checkpoint.
ClickHouse 26.4 is here! In this release, more features become SQL compatible, COUNT DISTINCT gets faster, EXPLAIN gets even prettier, and more
This post introduces Quack, the new client-server protocol for DuckDB. It explains the motivation for a client-server architecture and outlines the design considerations for the Quack protocol, including security, efficiency, and extensibility.
The title suggests a discussion on managing data assets at Netflix's scale.
Avride replaced Apache Iceberg with ClickHouse Cloud, cutting index lookup latency from 20 seconds to under 100ms and ingestion from hours to seconds.
The article focuses on the design and implementation of Pinterest's MCP ecosystem, outlining the key elements required for its successful operation.
The database industry is repeating a historical cycle where specialized systems create fragmentation that demands convergence. As AI agents become primary data consumers, organizations face a new challenge: context silos, where information exists but cannot be retrieved fast enough for autonomous sy
This post details Pinterest's approach to enhancing ad relevance by integrating real-time contextual information into their sequential recommender models. It explains the methods and architectural considerations involved in processing and utilizing real-time data for these systems.
The title suggests a discussion on scaling ArchUnit using Nebula ArchRules.
The article describes Discord's approach to automating the setup and management of ScyllaDB clusters at scale. It explains the challenges faced when configuring and operating dozens of database nodes and the solutions implemented to streamline this process, significantly reducing deployment time.
Apache Doris 4.1 introduces mature spill-to-disk capabilities, enabling Hash Join, Aggregation, and Sort operators to write intermediate state to disk when memory pressure rises so that memory-intensive analytical queries complete without OOM errors.
The program for DuckCon #7 Amsterdam, a DuckDB user conference, has been announced. The event will be held on June 24, 2026, and will run from 15:00 to 20:00 CEST.
This is Part 1 of a 15-part Apache Iceberg Masterclass. This article covers the fundamental question: what problem do table formats solve, and why does the choice between them matter? A data lake without a table format is a collection of files. It has no concept of a transaction, no mechanism to pre
This article presents container design patterns categorized by their coordination scope, providing a structured overview of common practices for distributed systems.
The article posits that many agent reliability issues stem from underlying data engineering problems.
The post discusses using ClickHouse as a Kafka sink, focusing on how async insert mode helps with high message rates but has buffering and dedupe behaviors that aren't always obvious.
Parloa leverages OpenAI models to power scalable, voice-driven AI customer service agents, enabling enterprises to design, simulate, and deploy reliable, real-time interactions.
This Anyscale blog post discusses the architecture of AI agents on Ray Serve, covering both single-agent and multi-agent architectures. It explores how to build and deploy AI agents using Ray.
Neon decoupled storage and compute to deliver up to a 5x performance increase on write-heavy workloads by disabling full-page writes.
DuckDB's Delta Lake and Unity Catalog extensions are no longer experimental. The post details the progress of these extensions.
Adding a column to a large production table used to require a plan involving migration scripts, maintenance windows, and backfill jobs that rewrite every data file to include the new column. Iceberg default column values eliminate the need for backfills during schema evolution.
The post details improvements to vLLM, focusing on correctness before corrections in reinforcement learning.
On May 5, 2026, DENIC published broken DNSSEC signatures for the .de TLD, making millions of domains unreachable. Here's what 1.1.1.1 saw, how serve stale cushioned the impact, and how we restored resolution.
How ClickStack and Odigos eliminate observability gaps with zero-code eBPF instrumentation and full-fidelity distributed tracing at scale.
This post links to a video demonstrating the DuckLake workflow using Rosetta DBT Studio. The presentation covers creating and importing lakehouse instances, exploring metadata, running SQL queries, and building reusable SQL Notebooks.
Agentic analytics makes query-readiness a write-side cost problem. This post compares Snowflake and ClickHouse under continuous ingest, showing how ClickHouse obtains query-ready data at 22× lower cost and delivers 31× better write-side cost-performance.
Singular Bank built Singularity, an internal assistant using ChatGPT and Codex to help bankers save 60–90 minutes daily on meeting prep, portfolio analysis, and follow-up.
Uber uses OpenAI to power AI assistants and voice features that help drivers earn smarter and riders book faster across a global real-time marketplace.
Simon Willison describes how he used AI agents to launch and run a cafe in Stockholm, detailing the architecture and lessons learned.
The article compares three patterns for integrating AI inference with Apache Kafka: external RPC, embedded, and sidecar, focusing on avoiding consumer rebalances, cutting costs, and scaling LLM pipelines.
This article explores the challenges and solutions for establishing reliable monitoring systems in a large-scale production environment. It details architectural considerations and best practices for ensuring consistent and accurate observability data.
Jikkou 1.0 is out, featuring Apache Iceberg integration for declarative management of namespaces, tables, and views. It also includes multi-cluster orchestration and Confluent Cloud RBAC.
StarTree claims a 5-37x performance improvement for queries on Iceberg tables compared to Trino and ClickHouse, based on their benchmark.
This article presents a self-healing layer designed to detect and correct hallucinations in RAG systems before they reach users by addressing issues in reasoning.
We use clickhousectl to spin up multiple ClickHouse versions side by side and benchmark two recent performance improvements.
OpenAI introduces MRC (Multipath Reliable Connection), a new supercomputer networking protocol released via OCP to improve resilience and performance in large-scale AI training clusters.
This article compares 'Realtime' and 'Pipelines' data processing approaches, both leveraging Postgres logical replication. It explains how these two methods address different problems and guides users on selecting the right solution for their use case.
From spinning disks to CPUs to cloud object storage, shifting bottlenecks have shaped Redpanda's architecture. Here’s what Cloud Topics revealed about today’s demand for high-latency storage.
OpenAI and PwC are collaborating to help businesses automate finance workflows using AI agents, with the aim of improving forecasting, strengthening controls, and modernizing the CFO function.
Figma details the architecture and implementation of PGKeeper, their custom Postgres connection pooler, explaining why existing solutions like PgBouncer didn't meet their needs.
How Qonto uses ClickHouse Cloud to power observability at scale — replacing sampling and hour-capped queries with two-week query windows, 99.84% compression on high-cardinality data, and an AI incident companion built on the ClickHouse MCP server.
This blog post discusses the simplicity of the DuckLake specification for dataframes.
OpenAI describes how it rebuilt its WebRTC stack to power real-time Voice AI with low latency, global scale, and seamless conversational turn-taking.
Icestream is a project enabling efficient streaming writes in Apache Iceberg.
A developer is building a no-code agent builder that uses DuckDB to create conversational analytics agents. These agents can respond to queries with interactive charts and UI, allowing users to query databases more easily.
This post proposes a framework for leveraging AI, emphasizing context as infrastructure, taste as configuration, verification for autonomy, scaling through delegation, and closing feedback loops for continuous improvement.
A user shares a community extension for DuckDB called `ducksmiles` that enables processing chemistry data, including SMILES, InChI, and PDB formats, directly within DuckDB. The extension supports functions like `mol_formula` and `mol_weight` and integrates with `read_csv_auto`, `read_text`, and `htt
Cloudflare completed an engineering effort to make its infrastructure more resilient using tools like Snapstone and the Engineering Codex. They implemented safer configuration changes and automated best practices to prevent future incidents.
This post from Pinterest Engineering focuses on optimizing network efficiency for machine learning workloads, presenting the first part of their strategy. It introduces and explains the 'Feature Trimmer,' a component designed to reduce network overhead in ML systems.
Fivetran accelerated the transpilation of SQL dialects in SQLGlot by compiling it with mypyc, resulting in faster translation between different SQL dialects for query engines.
This arXiv paper introduces a method to improve Text-to-SQL accuracy by using template-constrained decoding, particularly for recurring questions. It addresses challenges in real-world deployment of Text-to-SQL models, especially in complex or unseen schemas.
DuckDB infers NULL-only columns as the generic JSON type, causing staging issues when real values appear later. The solution involves using a synthetic canonical sample to ensure correct type inference from the outset.
This post discusses the shift of AI engineers from using frameworks like LangChain to building native agent architectures for production LLM applications.
The article explains how LanceDB scales vector search to 10 billion vectors and beyond. It covers the application of distributed indexing, distributed query execution, HNSW centroid routing, and fast RaBitQ rotation.
The post discusses the timeline, root cause, and fixes behind "goblin outputs," which are personality-driven quirks in GPT-5 behavior.
This post links to a DuckDB extension for vector search indexes with pluggable quantization.
Stripe introduces Link’s wallet for agents, offering programmatic access to generate one-time-use cards or Shared Payment Tokens, built on Stripe’s new Issuing for agents.
A user shares a practical troubleshooting guide derived from recurring Kafka production failures, covering issues like consumer lag, producers writing without consumers reading, and duplicate processing after restart due to offset commit problems.
This article details the development of Skipper, Airbnb's custom-built embedded workflow engine. It covers the architectural decisions, design principles, and operational experiences involved in creating a specialized orchestration solution for internal use cases.
The article explains the redesign of blob storage within the Lance format to elevate multimodal data to a first-class citizen. It outlines four storage semantics (Inline, Packed, Dedicated, External) that automatically adjust to different workloads.
This article explores Stripe's Radar system for detecting fraudulent transactions within 100 ms, detailing the architectural decisions behind its effectiveness.
The article explores how Lance can serve as a foundation for AI systems utilizing single-cell genomics atlases, paving the way for a new generation of biological modeling. It discusses the technical aspects of a storage format designed for these applications.
This post describes the architecture Pinterest implemented for its shopping conversion candidate generation process, tracing the journey from user clicks to identifying potential conversions. It outlines the system design and components involved in this complex data pipeline.
The post discusses how Iceberg deletion vectors offer a more efficient way to handle row deletions in data lakehouses, where deleting rows can be an expensive operation due to the immutable nature of Parquet files.
turbopuffer now allows combining attribute values into the scoring function of text queries. Ranking by attribute helps achieve better relevance in the first stage with the same scalability characteristics as BM25.
Choco used OpenAI APIs to streamline food distribution, boost productivity, and unlock growth, providing a customer story on the real-world impact of AI.
The article outlines a strategy for modernizing data platforms by migrating to an Apache Iceberg lakehouse. It suggests avoiding long ETL pipeline builds and focusing on faster time-to-value for analysts.
Pgrx is a framework for building PostgreSQL extensions using Rust, enabling developers to leverage Rust's safety and performance features within the Postgres environment.
Recent pg_clickhouse releases introduce JSONB, date/time, and array function pushdown, plus HTTP result set streaming for lower memory usage.
The article discusses using whisper.cpp within DuckDB to translate speech to text.
The paper introduces SQLyzr, a benchmark for evaluating text-to-SQL models, which have improved due to large language models. The platform addresses shortcomings in existing benchmarks.
The paper argues that the dominant approach in agentic AI, where large language models orchestrate information access by dynamically selecting tools, is misguided. It proposes an alternative architecture focused on data.
The paper studies the efficiency of loading and storing data in Apache Hudi, Apache Iceberg, and Delta Lake using Apache Spark.
The post discusses DeepSeek-V4, a model with a million-token context that agents can use.
ClickHouse positions itself as an alternative to Elasticsearch for log analytics by combining full-text search and large-scale analytics. The post alludes to a performance benchmark.
The article highlights a significant security vulnerability where misconfigured RAG pipelines are exposing vector databases to the public internet. A live map visualizes the scale of the leak, emphasizing the failure of perimeter security in the AI space.
The article describes how Spotify used Honk, Backstage, and Fleet Management to ease the pain of migrating thousands of datasets.
The article introduces the new GizmoSQL iOS app, which uses DuckDB.
DuckDB 1.5.2 is a new release of the SQL database that runs on laptops, servers, and in the browser.
A solution architect explains the architecture decisions critical for successful dbt implementation on Databricks, focusing on potential pitfalls and solutions.
The article discusses the need for both semantic and operational context for AI agents to trust data. It presents how Dagster and Atlan can provide both halves of the context layer.
ClickHouse reports a 30–55% speedup in ClickBench queries and ~15% reduction in compute costs on Google Cloud by migrating to Axion C4A instances.
The author compares different methods for efficiently modifying DuckDB from Java, highlighting performance improvements with the new UDF feature of the Java Drivers.
This article details the architecture and implementation of Airbnb's fault-tolerant system for storing operational metrics. It discusses the design choices made to ensure data durability, high availability, and scalability for critical observability data.
The article discusses how using dbt's semantic layer provides a foundation for building reliable and governed agentic analytics workflows.
Transient is a CLI tool to provide a governance layer for AI agents, including permission policies and auditing. It helps answer the question of what an agent did, whether it was authorized, and if it can be proven. The tool wraps the agent process and installs quickly.
ClickHouse Cloud now shards indexes across replicas, which distributes memory usage and improves performance for petabyte-scale workloads. This change reduces memory usage, speeds up index analysis, and improves performance.
Apache Arrow version 24.0.0 has been released with 259 resolved issues from 57 contributors. The announcement provides a link to the installation page.
This Redpanda blog post introduces Shadow Linking, a feature designed to simplify disaster recovery through real-time replication.
Lakeflow Jobs cannot see across workspace boundaries. This post explains how Dagster unifies multiple Databricks workspaces into a single asset graph with real dependencies.
This post from Pinterest Engineering details their approach to smarter URL normalization and content deduplication at scale. It explains how their proprietary MIQPS system contributes to efficiently identifying and handling duplicate content.
This article examines how GitHub built a security architecture that assumes the agent is already compromised.
Nava migrated ELO's payments monitoring platform from Elasticsearch to ClickHouse, cutting storage from 12 TB to 2 TB, slashing annual infrastructure costs by 87%, and delivering sub-2-second end-to-end latency across 300 real-time dashboards.
This article discusses lineage at the row level for Iceberg tables to track how specific rows were affected.
The article presents EvoRAG, a Knowledge Graph-based Retrieval-Augmented Generation framework designed to improve LLM reasoning by retrieving multi-hop paths from knowledge graphs. The framework aims to address the underperformance of existing KG-RAG solutions in real-world scenarios through feedbac
The article explores agentic visual analytics systems, where LLM-driven agents autonomously manage the full visual analytics pipeline. This approach seeks to shift users away from low-level tool manipulation towards higher-level, task-oriented interactions.
Explore the end-to-end pipeline of TurboQuant, a novel KV cache quantization framework. This overview breaks down how multi-stage compression achieves near-lossless storage through PolarQuant and QJL residuals, enabling massive context windows with minimal memory overhead
Hey folks. I find Online Machine Learning (OML) particularly appealing in data streaming environments, even though it hasn't yet seen widespread application across many domains. I wanted to build a complete Event-Driven Architecture that applies stateful stream processing to a real-world physical pr
The article discusses a scenario where a RAG system retrieves the right data but still produces wrong answers and offers suggestions to fix this issue.
The article suggests using Git worktrees to provide AI agents with isolated workspaces for parallel coding sessions, discussing the benefits and setup considerations.
An autonomous driving company consolidated fragmented data platforms by adopting Apache Doris as a unified analytics engine, enabling seamless search across text, vectors, labels, and metadata while reducing query times from minutes to seconds.
Brazilian fintech Trio cut storage by 88% and achieved a "generational leap" in speed by building a unified payment analytics platform on ClickHouse Cloud, handling 243M+ payments and 1B+ daily events with a sliding window approach for late and duplicate
pgvector is an open-source PostgreSQL extension that adds the ability to store, index, and search over vector embeddings, enabling similarity search and other vector-based operations directly within Postgres.
Agents of the Alley is a Context Engineering OS for Claude Code Agents, available on Github.
The article details five practices for reducing Kafka Streams rebalancing issues when running stateful processors with RocksDB on Kubernetes, including static membership configuration and session timeout tuning.
A Go package allows mounting fs.FS as a virtual file system in DuckDB.
Meta's Capacity Efficiency Program uses an AI agent platform to automate the identification and resolution of performance issues across its infrastructure. The platform leverages encoded domain expertise and a standardized tool interface to improve efficiency.
The WarpStream MCP Server allows AI assistants to connect to WarpStream clusters for querying logs, diagnosing issues, and inspecting ACL events directly from the IDE.
Meta shares lessons learned from their post-quantum cryptography (PQC) migration to assist other organizations in strengthening their resilience during the transition to post-quantum cryptography standards. They propose the idea of PQC Migration Levels to help teams manage the complex migration proc
The article introduces memweave, a system for agent memory that uses Markdown and SQLite, eliminating the need for a vector database.
Cloudflare's Artifacts provides Git-compatible versioned storage for code and data, designed for agents, developers, and automations. It supports creating millions of repos and forking from any remote.
This post shares a real-world story from Pinterest about diagnosing and resolving critical CPU bottlenecks discovered within their systems. It describes the investigation process to uncover these 'zombie' processes and the strategies implemented to mitigate them.
The article proposes disaggregated LLM inference, where prefill (compute-bound) and decode (memory-bound) operations are handled by different GPUs, potentially leading to 2-4x cost reductions.
Learn how ClickHouse uses primary indexes, lightweight projections, and skip indexes to prune data before reading it. Demonstrated on a 243 million row UK property sales dataset.
The article outlines design considerations and patterns for implementing agentic workloads on Apache Airflow. It covers principles to ensure these systems are observable, retryable, and auditable in production environments.
The article describes a system built with Apache Airflow 3 to analyze large survey datasets using AI. It enables users to pose natural language questions, which the system converts to SQL queries, executes, and returns results.
Most RAG tutorials focus on retrieval or prompting, but the real problem starts when context grows. This article outlines a context engineering system built in pure Python that controls memory, compression, re-ranking, and token budgets to keep LLMs stable under realistic constraints.
This article details the architectural and engineering approaches Airbnb uses to build privacy-first data systems that empower social experiences. It covers the design principles and technical implementations ensuring user privacy while fostering connections on the platform.
An iOS app, GizmoSQL, runs DuckDB as a server with Arrow Flight SQL and supports mounting a DuckLake, enabling a "data lake in your pocket." The app runs the TPC-H 1GB benchmark in under 2 seconds on an iPhone 16 Pro Max.
This article discusses the evolving landscape of AI agent frameworks and harnesses, suggesting that while frameworks might be becoming cheaper, the underlying need for structured agent orchestration remains.
This article explains how to optimize GPU efficiency, covering GPU architecture, performance bottlenecks, and optimization strategies using PyTorch and custom kernels.
The article discusses context graphs as a crucial layer between human decisions and AI agents.
Mintlify replaced PostHog with ClickHouse Cloud, resulting in faster dashboard load times, no rate limit errors, a 30% NPS improvement, and a 60% cost reduction.
A developer built a browser-based spreadsheet diff tool using DuckDB WASM to compare Excel/CSV files, highlighting differences in 42k rows × 14 cols in approximately 3 seconds without a server. The tool handles date formats, floating point noise, and case inconsistencies.
This blog post details how agentic AI can be used to automate demand forecasting in CPG supply chains, potentially improving accuracy and efficiency while reducing manual work.
This Reddit post discusses DuckDB's architecture, comparing Ducklake to Snowflake and highlighting potential drawbacks of using object stores for metadata like Iceberg does. The post explores what differentiates MotherDuck's technical architecture from Snowflake's.
The article introduces the `apache-airflow-providers-common-ai` package, which brings native LLM and AI agent capabilities directly into Apache Airflow. This provider package offers direct integration rather than wrapping external frameworks.
This post details Pinterest's strategy for scaling their recommendation systems through the implementation of request-level deduplication. It explains the architectural considerations and benefits of this optimization technique for high-throughput ML serving.
In this article, we will look at how the LinkedIn engineering team rebuilt the Feed and the challenges they faced.
DuckLake v1.0 has been released.
Cloudflare introduces Durable Object Facets, which allows Dynamic Workers to instantiate Durable Objects with isolated SQLite databases, enabling platforms that run persistent, stateful code generated on-the-fly.
Cloudflare Sandboxes provide AI agents with persistent, isolated environments, including a shell, filesystem, and background processes.
Outbound Workers for Sandboxes provide a programmable, zero-trust egress proxy for AI agents. This allows developers to inject credentials and enforce dynamic security policies without exposing sensitive tokens to untrusted code.
DuckDB v1.5.2 is a patch release. DuckLake is released.
The article announces the release of DuckLake 1.0.
This post discusses a custom-built solution for Kafka Streams state store recovery during disaster recovery, addressing a previously unsolved problem in the ecosystem.
Most ReAct-style agents are silently wasting their retry budget on errors that can never succeed. In a 200-task benchmark, 90.8% of retries were spent on hallucinated tool calls — not model mistakes, but architectural flaws. This article shows why prompt tuning won’t fix it, and the three structural
Walkthrough of querying data lake files with DuckDB, covering Parquet, Iceberg, and S3 integration patterns.
This LangChain blog post discusses the growing importance of agent harnesses in building AI agents and their connection to agent memory. It highlights the potential drawbacks of using closed harnesses, particularly those behind proprietary APIs, which can limit control over the agent.
LineageScope is a static analyzer for SQL, dbt, Airflow, Spark, and data contracts. It aims to provide data lineage and enforce data contracts.
The article argues that AI coding assistants require a persistent memory layer to overcome the limitations of stateless LLMs. This memory layer improves code quality by providing systematic context across sessions.
This Netflix tech blog post is about evaluating show synopses with LLM-as-a-Judge.
This post announces the open-sourcing of a formally verified TLA+ specification for a leaderless log protocol for Kafka, highlighting the discovery of a design bug through verification. It also mentions using Claude Code to generate a working Rust implementation from the specification, demonstrating
This article from the DuckDB website discusses the design and implementation of DuckDB internals, which is useful for understanding its architecture and performance characteristics.
This article discusses context engineering techniques for AI coding agents, specifically focusing on Claude code sub-agents. It explores how to structure prompts and context to improve the performance of AI coding assistants.
This article outlines golden rules for agent-first product engineering. It explores considerations for designing products around AI agents and their capabilities.
Meta shares its approach to modernizing WebRTC, the technology powering real-time audio and video across their platforms. The article highlights the challenges of forking a large open-source project and how Meta addressed them to stay aligned with community upgrades.
Deep Agents deploy is being launched in beta as a way to deploy a model agnostic, open source agent harness in a production ready way. It is built on Deep Agents and designed for an open world.
This Reddit post links to a DuckDB tutorial demonstrating how to build a text-to-SQL agent that can automatically recover from bad SQL queries. The approach leverages a tool-calling agent loop to inspect and correct errors.
This paper presents LASER, a data-centric method for low-cost and efficient SQL rewriting based on SQL-GRPO, aimed at transforming queries into more efficient variants for database optimization.
This paper introduces AV-SQL, a method for decomposing complex Text-to-SQL queries using agentic views, aimed at improving the accuracy and efficiency of natural language to SQL translation.
This paper introduces SQLStructEval, a framework for evaluating the structural reliability of LLM-generated SQL queries, investigating the structural behavior of these queries to determine if they are sound.
This article draws parallels from Chipotle's operational model to discuss real-time data product architectures. It explores a third option that balances fresh data and fast queries, applicable to building data products for modern applications and AI agents.
Redpanda Connect now offers native CDC for Oracle, enabling real-time data access without requiring rearchitecting. The solution eliminates the need for a JVM, middleware, and related operational overhead.
This post recaps ClickHouse's involvement at FOSDEM 2026 in Brussels. It highlights the community's activities during the event.
This post highlights GlassFlow's work on achieving high-throughput (500k+ events/sec) transformations for ClickHouse ingestion, particularly in observability and real-time analytics pipelines. It addresses challenges related to scaling throughput.
This Confluent blog post details how Agent Taskflow built a production-grade AI orchestration platform using Confluent and AWS, avoiding self-managed Kafka. The article focuses on scalability and the benefits of using managed services.
This post from Pinterest Engineering explores their initiatives and approaches to improving overall system performance for a wide range of users and services. It likely covers methodologies, tooling, or cultural shifts to foster a performance-first mindset.
This blog post argues that retention limits, sampling, and metric roll-ups hinder AI-driven observability workflows. It suggests that these practices are inadequate for handling the demands of full-fidelity data required by modern AI.
Cloudflare's blog post details how they automated the generation of malware trigger packets using symbolic execution on BPF bytecode. By leveraging the Z3 theorem prover, they significantly reduced analysis time, improving their ability to detect and respond to threats.
This blog post analyzes DuckLake's claim of a 900x speed improvement by inlining data into the catalog, a topic of interest for data engineers optimizing query performance.
This paper proposes a "Black-Hole Attack" against vector databases, where malicious vectors injected near the geometric center can compromise retrieval. It highlights the need for security considerations in vector database design.
This paper introduces Cortex AISQL, a production SQL engine from Snowflake that integrates native semantic operations directly into SQL. This allows users to combine relational operations with semantic reasoning for querying unstructured data.
The article discusses effective strategies for managing the context window in AI agents. It emphasizes improving performance, reducing costs, and maintaining relevant outputs, which is valuable for optimizing AI systems.
This post describes how to combine Fivetran, DuckDB, and Claude to enable conversational analytics on a data lake, highlighting the potential for interacting with data through natural language queries using an MCP server.
This article details the construction of Airbnb's high-volume metrics pipeline, outlining the integration of OpenTelemetry and vmagent. It covers the architectural considerations, implementation specifics, and operational insights for processing vast amounts of observability data.
This post outlines the evolution of multi-objective optimization techniques implemented for the Pinterest Home feed, tracing how these complex systems have developed over time. It describes the challenges and solutions in balancing multiple optimization goals for user experience.
The PFC-JSONL extension, which enables block-level timestamp filtering for compressed JSONL logs, has been merged into the DuckDB Community Hub. This allows users to query compressed JSONL files more efficiently using DuckDB.
The post describes a hybrid PyMuPDF and GPT-4 Vision pipeline that significantly reduced the time required for document extraction from PDFs, replacing manual effort, and discusses why the latest models weren't always the optimal solution.
This article delves into the design of an AI agent for navigating large-scale event data, focusing on transforming query patterns into intelligent tools and crafting an effective agent architecture, which offers practical insights into building agents for complex data environments.
This article discusses techniques for optimizing context, a finite resource, when designing AI agents, focusing on how to best utilize available information to enhance agent performance.
ClickHouse version 26.3 introduces async inserts by default, improved JOIN reordering, and materialized CTEs. These features could improve query performance and data management for users.
The Apache Arrow team announced the version 23 release of the Apache Arrow ADBC libraries, which includes 41 resolved issues from 20 contributors. This release focuses on the libraries, which are at version 23, with the API specification versioned separately.
The article announces the release of Apache Airflow 3.2.0, focusing on data-aware workflows. Key features include asset partitioning for granular pipeline orchestration and support for multi-team deployments at enterprise scale.
This Netflix tech blog post discusses interval-aware caching for Druid at scale. The article likely contains valuable insights into Druid performance and optimization strategies.
Enterprise AI SaaS automates customer enablement with a 5-agent workflow to close adoption gaps, reduce churn, and scale training across industries
Meta shares how they used AI to map tribal knowledge within large-scale data pipelines, highlighting the challenges and limitations of AI coding assistants' understanding of complex codebases.
This ByteByteGo article explores context engineering for LLMs, explaining how LLMs process information and outlining strategies to improve context utilization.
Respan AI migrated to ClickHouse Cloud for high-throughput LLM observability after outgrowing Postgres. The company now processes 50 million daily events using incremental materialized views.
This article discusses the author's experience building with AI over three months after eight years of wanting to, likely covering lessons learned and insights gained.
This LangChain blog post discusses continual learning for AI agents, highlighting that learning occurs at the model, harness, and context layers, not just model weight updates. Understanding these distinctions is crucial for building systems that improve over time.
The article introduces a Syntaqlite Playground, which is related to dbxlite and Metastax. It's a useful tool for Staff+ level data engineers, ML engineers, and analytics practitioners.
Netflix details how they are using multimodal intelligence to improve video search capabilities. The article likely covers the engineering challenges and solutions involved in building and deploying such a system at scale.
This dbt blog post discusses how to operationalize analytics agents by building context for LLM models using dbt and MCP servers. It explores practical ways to integrate LLMs into analytics workflows.
This article describes how a glibc update in 2018 silently invalidated Postgres text indexes, leading to incorrect query results. It highlights the importance of understanding collation settings for maintaining data integrity in Postgres.
This post details a self-healing deployment pipeline for a GTM Agent. The system automatically detects regressions after each deploy, determines if the change caused the regression, and uses an agent to create a pull request with a fix, minimizing manual intervention.
Atlan describes their migration of workflow orchestration from Argo to Temporal. The post details the reasons for the migration, the crossover architecture used during the transition, and lessons learned from rebuilding orchestration in production.
This paper critiques the use of average recall as the dominant metric for evaluating vector databases, which are crucial in AI systems. It argues that relying solely on average recall can be problematic for users and researchers optimizing these systems.
This paper discusses DocETL, a declarative system for LLM-powered data processing that has gained traction across various domains. DocETL allows users to define complex data processing pipelines using LLMs, enabling tasks like information extraction and data transformation from unstructured document
The agent security market is rapidly developing with companies offering runtime identity enforcement and permission revocation for autonomous agents.
ClickHouse details their work on agentic coding. The article likely details the practical implementations and potential benefits of this approach within the ClickHouse ecosystem.
Netflix details the implementation of Variable Bit Rate (VBR) for all their live events to improve streaming quality.
This article describes how to use Dagster's dbt integration to run and monitor dbt models as part of a larger asset-driven pipeline, focusing on lineage and scheduling improvements.
Learn step-by-step how to debug Dagster pipelines directly inside Docker, bridging development and deployment environments with practical tools.
Warm Docker containers reduce cold starts, learn how Dagster Cloud deploys 5x faster using PEX-based builds.
Use proven tips to make your Python code faster and more efficient, especially for data engineering and pipeline-heavy workloads.
Meta details KernelEvolve, a Ranking Engineer Agent used to optimize AI infrastructure for ads ranking, focusing on autonomous design, execution, and analysis of ranking model experiments.
LangChain reports that open models like GLM-5 and MiniMax M2.7 are now comparable to closed frontier models on agent tasks like file operations and tool use. The article presents evaluation results and instructions for using these open models.
Cloudflare discusses the challenges and opportunities in cache design presented by the explosion of AI-bot traffic, detailing the differences between AI bot traffic and human traffic and providing some early ideas for system design.
This article discusses how data leaders should design the interface between data platforms and the teams that rely on them. It emphasizes the importance of clear boundaries and well-defined responsibilities in data platform engineering.
Redpanda details how they used profile-guided optimization to improve performance and reduce latency in Redpanda 26.1. The article offers a behind-the-scenes look at their optimization process.
This blog post from the DuckDB team introduces data inlining in DuckLake to enable streaming for data lakes. It details the motivation, implementation, and benefits of this approach, including improved performance and reduced latency.
Rill leverages ClickHouse to deliver real-time operational BI for over 100 billion daily events. The integration offers instant data exploration and conversational analytics via a declarative, BI-as-code workflow.
MotherDuck explores the performance and challenges of working with large datasets in Redshift.
Dagster 1.12 introduces a redesigned UI, Components GA, streamlined deployment workflows, and major orchestration upgrades. These enhancements aim to make data orchestration faster, simpler, and more reliable for users.
Gradient Labs is deploying AI account managers for banks using GPT-4.1 and GPT-5.4 models. These agents aim to automate banking support with low latency and high reliability.
This blog post explains multimodal embeddings for searching across different data types (text, images, audio, video) in RAG systems. It provides practical implementations using Weaviate and Gemini.
This DuckDB blog post humorously explores an alternate reality where Dutch, not English, became the dominant language for SQL. It poses the question of how this linguistic shift might have shaped the development and standardization of SQL.
The Airbyte blog discusses why LLM agents often struggle to move beyond basic text-to-SQL functionality. It argues that a missing context layer is crucial for enabling truly intelligent, action-driven AI systems.
The article introduces Dux, distributed DuckDB-backed dataframes on Apache Beam. This is a .
ClickHouse has announced the general availability of its Bring Your Own Cloud (BYOC) offering on Google Cloud. This allows users to run ClickHouse within their own Google Cloud account while maintaining full data sovereignty and zero-trust networking.
Meta is scaling its Ads Recommender runtime models to LLM-scale and complexity using the Meta Adaptive Ranking Model. This article discusses how they are bending the inference scaling curve to deliver better experiences for people and better results for advertisers using AI recommendation systems.
This blog post from AWS covers how to give Kafka clients in different VPCs and AWS accounts secure private access to MSK Serverless clusters. The solution addresses the limitation of MSK Serverless supporting PrivateLink connectivity for up to 5 VPCs in the same account.
This article details the data schema, embeddings, and graph design for an agentic query engine on ApertureDB, focusing on managing mixed data types. It is a .
This paper introduces Exqutor, an extended query optimizer designed for vector-augmented analytical queries, particularly in Retrieval-Augmented Generation (RAG) pipelines. It aims to improve the efficiency of retrieving relevant external knowledge for large language model inference.
Padlet leverages ClickHouse Cloud to provide real-time analytics for classrooms, processing billions of events per month with sub-second query performance without requiring a dedicated data team, showcasing ClickHouse's scalability and ease of use.
This post introduces kernel-anvil, a tool for auto-tuning llama.cpp kernels on AMD GPUs, resulting in a 2x decode speedup by optimizing kernel configurations per model shape. The tool profiles GGUF model layer shapes and generates optimal kernel configs loaded at runtime, without recompilation.
This article describes the architecture of Redpanda Cloud Topics, a new replication mechanism that uses object storage to reduce costs. The discussion of internals is valuable for engineers working with streaming data.
This article explains how to use HNSW indexes effectively with JOINs and WHERE clauses in DuckDB, demonstrating how to combine approximate nearest neighbor search with standard SQL operations for efficient data retrieval.
The article describes a technique for creating self-healing neural networks using PyTorch. It demonstrates how to detect model drift and adapt in real time without retraining.
This WarpStream blog post explains Kafka transactions and contrasts them with WarpStream's implementation, which eliminates stateful brokers. It highlights how WarpStream achieves the same protocol guarantees with different internal architecture.
WarpStream demonstrates a 4x reduction in cloud infrastructure costs compared to self-hosted Kafka in their benchmarks. The savings are attributed to eliminating inter-AZ fees and replacing EBS with object storage.
This Neon blog post discusses their approach to zero-downtime patching using prewarming techniques to ensure continuous availability of customer databases. It details their system's redundancy and failover mechanisms.
The LangChain blog post offers a checklist for evaluating AI agents, covering error analysis, dataset construction, grader design, and offline/online evaluation. The checklist is intended to help ensure production readiness.
This Databricks blog post discusses techniques for ensuring database availability during patching in Lakebase. It focuses on prewarming as a method to minimize downtime during updates, which is crucial for maintaining service reliability in data platforms.
This post shares the configurations used to push Qwen 3.5 27B to 1,103,941 tok/s on 12 nodes with 96 B200 GPUs using vLLM. The improvements came from changes to DP, context window, FP8 KV cache, and MTP-1 speculative decoding.
This paper introduces the concept of Agentic Context Engineering for self-improving language models. It is focused on improving the performance of LMs through dynamic context management, which is relevant to AI agents and context-aware data systems.
Vizier is a physical design advisor for DuckDB. It analyzes queries and recommends changes to the database layout, such as sort orders and indexes, to improve performance.
This article presents ten best practices for ClickHouse, covering topics like primary key design, data types, materialized views, and join optimization. Benchmarks on a 150M row dataset illustrate the impact of these practices.
Cloudflare describes a Kubernetes fix involving fsGroupChangePolicy that reduced Atlantis instance restart times from 30 minutes to 30 seconds by addressing a bottleneck in volume permission handling.
ClickHouse now supports direct querying of Iceberg and Delta Lake formats across major cloud catalogs. This feature eliminates the need for data migration, improving data lake accessibility.
A physical design advisor called Vizier has been developed for DuckDB. It analyzes queries and suggests changes to the database's physical layout, such as sort orders and indexes, to improve query performance.
This paper details ByteHouse, ByteDance's cloud-native data warehouse designed for real-time multimodal data analytics. It addresses the need for efficient and cost-effective data analytics infrastructures in handling the demands of intelligent data services within ByteDance's production environment
This blog post from Firetiger explores strategies for handling large tool results within AI agent workflows. It discusses approaches like summarization, pagination, and streaming to manage the volume of data returned by tools used by agents.
Pgsemantic is a new open-source project that enables instant vector search capabilities within a Postgres database. The tool aims to simplify the process of integrating vector embeddings for semantic search applications directly within Postgres.
Shared-nothing made sense when storage was slow, but shared storage flips that tradeoff. The architectural case for building Kafka directly on object storage.
Bento is the MIT-licensed open source fork of Benthos — stateless stream processing, no feature gating, no license traps, full connector ecosystem intact.
Kafka assumes stateful, partition-owning brokers. How WarpStream reverse-engineered it for stateless Agents. A deep dive into diskless Kafka load balancing.
Object storage does not GC itself; compaction and retention create orphaned data at scale. Five GC strategies for S3-native distributed systems compared.
When pprof hit its limits, WarpStream used gcore and viewcore to trace a goroutine leak in our control plane. A Go debugging guide for distributed systems.
Grafana Labs needed tens of GiB/s with zero inter-AZ costs for Cloud Metrics. How WarpStream diskless architecture met their scale without Kafka ops overhead.
Build real-time security threat monitoring with WarpStream, RisingWave, and Grafana: one materialized view per metric, no complex pipelines, no extra infra.
Cloud block storage costs up to 24x more per GiB than S3. The exact numbers behind why disk-based Kafka clusters drain your budget and what diskless changes.
WarpStream uses Antithesis to deterministically simulate its full SaaS, from signup to Kafka workloads, surfacing correctness bugs random testing misses.
WarpStream Orbit replicates any Kafka-compatible cluster offset-for-offset, preserving consumer groups, ACLs, and configs for zero-gap migrations and DR.
WarpStream BYOC needs zero cross-account IAM access. Raw data stays in your VPC, only metadata touches WarpStream. Secure by design for sensitive workloads.
Cluster quotas and mirroring do not fix Kafka noisy neighbors; they shift the cost. How WarpStream Agent Groups isolate workloads without dedicated clusters.
WarpStream Agents validate records against AWS Glue Schema Registry, blocking malformed data at ingest without dead-letter queues or extra infrastructure.
WarpStream natively embeds Bento inside Agents — Kafka Connect-style integrations and stream processing with zero additional infrastructure inside your VPC.
S3 API call costs can silently inflate your streaming bill. How WarpStream uses distributed memory-mapped caching across Agents to slash GET request volume.
How WarpStream built usage-based billing for its core product, separating events from metrics to keep pricing logic auditable and updatable post-facto.
WarpStream Managed Data Pipelines: fully-managed Bento inside your VPC — SaaS UX, data stays in your account, YAML-driven pipelines with rollback support.
How WarpStream implemented Kafka compacted topics with only 128 MiB RAM — tracking millions of dedup keys without traditional broker memory overhead.
How Goldsky scaled blockchain data to 100 PiB and 100K+ partitions on WarpStream for 10x cheaper than their previous Kafka vendor with zero bottlenecks. This production war story shows the benefits of WarpStream for high-volume streaming data.
Run WarpStream on Tigris for globally distributed, durable Kafka streaming. This setup eliminates region-specific bucket planning and hidden data transfer fees, offering a streamlined approach to managing streaming infrastructure.
WarpStream Multi-Region Clusters deliver RPO=0, ensuring that every acknowledged write survives full regional cloud outages without manual failover or tuning. This post highlights the robustness of WarpStream for mission-critical streaming applications.
Learn how to send structured .NET logs directly to ClickHouse using Serilog — with full schema control, full-text search, and SQL queries over your log data. This post provides a step-by-step guide for setting up and using the integration.
Learn how OpenAI’s Model Spec serves as a public framework for model behavior, balancing safety, user freedom, and accountability as AI systems advance. This post details the considerations and mechanisms used to ensure responsible AI deployment.
This article details an experience using AI for scientific research where the AI hallucinated data. It underscores the importance of verifying AI outputs, especially in data-driven fields.
This paper presents flexvec, a retrieval kernel that exposes the embedding matrix and score array as a programmable surface. It is designed for AI agents and offers opportunities to expose more of the retrieval pipeline to the caller.
A developer implemented ACORN for prefiltered approximate nearest neighbors on a fork of the DuckDB VSS extension, showing significant speedups over brute-force filtering on high-dimensional vector workloads.
This article explains how Materialize enhances query performance by compiling SQL's complex type system into simpler representation types. This process helps reduce unnecessary casts, enables more effective query optimizations, and generally increases processing efficiency.
OpenAI launches a Safety Bug Bounty program to identify AI abuse and safety risks, including agentic vulnerabilities, prompt injection, and data exfiltration. This program encourages community participation in enhancing the security and robustness of AI models.
Trino is adding support for the NUMBER data type to handle high-precision numeric types beyond the existing DECIMAL limit. This will allow Trino to query data from sources that use these types without loss of precision, improving interoperability.
ClickHouse Cloud's two-window recommender and target-tracking CPU algorithm cut scale-down latency from 30 hours to 3 hours while eliminating oscillations and reducing infrastructure costs. The post details the algorithm and its impact on autoscaling performance.
WarpStream Schema Linking mirrors Confluent-compatible schema registries into a BYOC instance, preserving schema IDs and compatibility rules for disaster recovery.
WarpStream's diskless, S3-native Kafka implementation aims to replace traditional broker-based streaming by eliminating disks, brokers, and inter-AZ costs.
This article answers questions about WarpStream's architecture, BYOC vs. Serverless options, pricing, Kafka compatibility, performance trade-offs, and zero-disk streaming.
After S3EOZ's 85% price drop, WarpStream benchmarks show 3x better latency vs. standard S3 at just 15% higher TCO. Full methodology and real workload numbers are presented, offering a practical comparison for those considering cloud-based streaming solutions.
WarpStream Diagnostics continuously scans clusters for cost inefficiencies and health issues, surfacing actionable fixes before they become incidents. This proactive approach helps data engineers maintain optimal performance and prevent costly disruptions in their streaming data pipelines.
WarpStream + Materialize centralize streaming business logic in SQL with dbt-style version control, enabling operational data products without brittle ETL pipelines. This combination offers a streamlined approach for building and managing real-time data applications.
Kafka idempotent producers without stateful brokers require rethinking deduplication. WarpStream uses retroactive tombstones to separate data from metadata, providing a technical solution for ensuring data integrity in streaming applications.
Tiered storage still runs stateful brokers with expensive disks and inter-AZ replication. It does not solve the real cost problem at the heart of Kafka, offering a critical analysis of a common architectural pattern.
OSS big data tools like Kafka were built for hyper-scalers, then given to everyone. The article discusses why on-prem assumptions in open source infra cause pain in the cloud, offering a high-level perspective on cloud infrastructure design.
Kafka offset-based consumer lag is misleading when message sizes vary. The post shows how to instrument time-based lag metrics for an accurate view of consumer group health, offering a practical solution for monitoring streaming applications.
The article provides a practical guide to Kafka disaster recovery and multi-region data sharing. It discusses how WarpStream Active-Active clusters achieve RPO=0 with no tuning required, showcasing an approach to building highly available streaming infrastructure.
Moda leverages a multi-agent system built on Deep Agents and LangSmith to enable non-designers to create professional-grade visuals. The article highlights a specific use case of AI agents in a design context.
This article outlines the architecture that allows Netflix to live stream to 100 million devices in 60 seconds. It focuses on the challenges and solutions involved in building a large-scale live streaming system.
This article focuses on offline evaluation frameworks for production-ready LLM agents. It tackles the challenge of proving that these complex systems work reliably before deployment.
Cloudflare introduces Dynamic Workers for executing AI-generated code in secure, lightweight isolates. This technique achieves millisecond startup times, significantly faster than traditional container-based sandboxing for AI agents.
The ClickHouse blog details the design of their new text index for high-performance full-text search, especially when data is stored in object storage. The post explains how the design maintains speed at scale.
Cogent Security explains how they built an AI-native vulnerability management platform using ClickHouse. They emphasize ClickHouse's speed as crucial for countering AI-enabled attacks.
This paper introduces BubbleRAG, an evidence-driven retrieval-augmented generation approach for black-box knowledge graphs. It aims to address the limitations of existing graph-based RAG approaches related to recall and precision.
Stripe Engineering details how Radar uses machine learning to prevent free trial abuse. The system predicts abusive behavior with 90% accuracy, based on common trial terms violations.
In this article, we will look at how agentic RAG works, how it improves upon standard RAG, and the trade-offs that should be considered.
Cloudflare's Gen 13 servers introduce AMD EPYC™ Turin 9965 processors and a transition to 100 GbE networking to meet growing traffic demands. In this technical deep dive, we explain the engineering rationale behind each major component selection.
Cloudflare’s Gen 13 servers double our compute throughput by rethinking the balance between cache and cores. Moving to high-core-count AMD EPYC ™ Turin CPUs, we traded large L3 cache for raw compute density. By running our new Rust-based FL2 stack, we completely mitigated the latency penalty to unlo
This arXiv paper presents a novel approach to stream processing by exploring functional isolation to reduce infrastructure costs. It discusses how concurrent workloads can extract insights from real-time data streams while optimizing resource utilization.
The paper introduces ReViSQL, an approach to translating natural language to SQL, aiming to achieve human-level performance. The research focuses on enhancing SQL reasoning by utilizing large language models and AI agents to decompose complex queries.
This paper provides a comprehensive survey on vector databases, covering storage and retrieval techniques, as well as challenges in the field. Vector databases are increasingly important due to their integration with large language models and applications in machine learning.
DuckDB 1.5.1 is released, including fixes and Lance support. The release notes are available on GitHub, and the new version can be installed from the installation page.
Trigger.dev describes their architecture for giving every user SQL access to a shared ClickHouse cluster. This pattern can be useful for connecting agents to platforms and building ETL systems.
This article uses compound probability to illustrate how seemingly accurate AI agents can fail in multi-step tasks. It also proposes a pre-deployment framework to mitigate such failures in production.
A Kafka Streams application with a large RocksDB state store (300M+ keys) experiences slow DR rebuild times (45+ minutes to 2 hours). The author is looking for solutions to improve rebuild performance, which is a common challenge in Kafka Streams.
WarpStream has added a built-in observability layer that stores structured operational events directly as Kafka topics in object storage, allowing users to query those topics.
This blog post describes WarpStream Events, an observability layer that captures Agent logs, ACL decisions, and pipeline execution logs, and allows users to search and visualize them with zero ops.
This article discusses potential failure modes in agentic RAG systems, including retrieval thrash, tool storms, and context bloat, and suggests methods for early detection to avoid high cloud costs.
This article presents a detailed example of how timely dataflow's approach to progress tracking can achieve orders of magnitude more efficiency than other stream processors. It explains the mechanisms behind achieving a 100x speedup.
Kwai, a short-video platform, unified its advertising analytics by migrating from ClickHouse and Elasticsearch to Apache Doris. This resulted in up to 90% latency reduction and a 3x increase in write throughput.
DuckDB has a flexible extension mechanism that allows extensions to be loaded dynamically at runtime, and this post shows how to build them in C#. This extension mechanism can add support for new file formats, introduce custom types, or provide specialized analytical functions.
OpenAI details their approach to monitoring internal coding agents using chain-of-thought analysis to detect and mitigate risks of misalignment, focusing on AI safety.
PondDB is presented as a self-hosted agent memory database built on DuckDB.
The article introduces Blobsearch, an Elasticsearch alternative based on object storage (like S3) and DuckDB for querying logs rapidly. It focuses on using a durable storage solution (S3 with Parquet) combined with the analytical capabilities of DuckDB for cost-effective log analysis and monitoring
A late-adopter walkthrough of using dbt with DuckDB for local development, covering the setup, tradeoffs, and workflow differences compared to cloud warehouses.
Friend bubbles in Facebook Reels highlight Reels your friends have liked or reacted to, helping you discover new content and making it easier to connect over shared interests. This article explains the technical architecture behind friend bubbles, including how machine learning estimates relationshi
This article explores using Apache Beam to ingest metrics into ClickHouse; this provides insights into how to leverage a data processing framework for efficient metric storage and analysis in a columnar database.
WarpStream eliminates disks, brokers, and inter-AZ costs by using an S3-native architecture. This article makes the case for this new approach to streaming.
Ingelt is a Rust/Axum gateway that parses 33 legacy industrial protocols inside 64MB WebAssembly sandboxes. The WASM isolation approach prevents malformed input from crashing the host process.
This post details how ClickStack integrates with ClickHouse to optimize queries for observability workloads. It covers techniques like progressive time window pagination, chunked charts, and automated use of materialized views, offering insights into performance tuning.
This Postgres Weekly article discusses how a badly written query caused an OOM (Out-Of-Memory) killer issue, even with ample RAM. The culprit was `work_mem` exceeding expectations; this is a cautionary tale regarding resource allocation and query optimization in Postgres.
Redpanda is open-sourcing their AI SDK for Go, designed for observable, resilient, and production-grade AI tooling.
Nemotron 3 Nano 4B is presented as a compact LLM suitable for local AI, offering an efficient option for running inference on resource-constrained devices. Staff+ ML engineers working on edge deployment or low-latency applications should investigate this model's architecture and performance characte
This article analyzes the characteristics of well-written corporate engineering blogs. It provides valuable insights for creating engaging and informative technical content within organizations.
ProtoScience is a deterministic pipeline designed to autonomously discover governing equations from raw numerical data. The system uses sparse regression and statistical validation without relying on LLMs.
The Get Shit Done framework uses meta-prompting and spec-driven development to structure LLM-powered system builds. It emphasizes generating detailed specifications before code, reducing iteration cycles.
This dbt blog post discusses how a semantic layer provides a consistent and governed foundation for AI systems used in analytics.
Meta’s Ranking Engineer Agent (REA) autonomously executes key steps across the end-to-end machine learning (ML) lifecycle for ads ranking models. This post covers REA’s ML experimentation capabilities: autonomously generating hypotheses, launching training jobs, debugging failures, and iterating on
The post discusses a shift towards autonomous, continuously evolving AI agents and integrations with tools like NVIDIA NemoClaw.
The author implemented a single binary solution to replace both Postgres and ClickHouse for web analytics.
This article explores context engineering, a topic critical for building AI-ready data systems. The post discusses designing data systems for AI consumption, machine-readable metadata, and contextual memory, providing insights into creating effective data pipelines for AI applications.
An introduction to data-centric query compilation, covering how modern engines like HyPer and Umbra generate machine code from query plans by pushing data through tight loops rather than pulling through iterator trees.
This article highlights the importance of extended statistics in Postgres for query optimization. It likely covers how to create and use extended statistics to improve query performance, especially for complex queries or datasets with skewed data distributions.
Reddit's engineering team migrated their petabyte-scale Kafka deployment from EC2 to Kubernetes. The article details the challenges they faced and the solutions implemented for a successful migration.
This Dagster blog post details CI/CD workflows using branch deployments, automatic retries, and backfill strategies; it also covers data quality via asset checks and monitoring with Dagster Insights, offering actionable advice for managing production data pipelines.
This article explores using DuckDB transpilation to reduce warehouse costs. It could involve techniques for rewriting SQL queries to leverage DuckDB's efficient execution or using DuckDB as a local processing layer before data warehousing, offering a practical method for cost optimization.
Covers subagent patterns for building composable AI agents that delegate tasks to specialized sub-agents, with practical implementation details.
Socialpruf replaced Neon with Postgres managed by ClickHouse, resulting in up to 5x faster query performance. The migration eliminated network transfer costs and improved the speed of real-time social analytics dashboards.
This article explores a neuro-symbolic system where a neural network learns fraud rules automatically, extracting IF-THEN rules during training; the experiment uses a hybrid neural network with a differentiable rule-learning module on the Kaggle dataset.
This article discusses building a product analytics warehouse directly on Postgres. The article likely details schema design choices, performance optimization strategies (indexing, partitioning), and extension usage (like pgvector) relevant for those using Postgres beyond traditional transactional w
The article compares Exasol, ClickHouse, StarRocks, Trino, and DuckDB across concurrency, data volume, and node scaling. While, the comparison could highlight architectural differences, performance trade-offs, and suitability for different analytical workloads across these popular SQL engines.
Flock v0.7.0 is a DuckDB extension that allows users to run RAG, Claude, and LLM metrics directly in SQL. The extension aims to eliminate the need to move data into Python scripts for semantic tasks.
Avalon Synthetic clinical data pipeline , generate realistic FHIR R4 patient data, normalize it through Forge, and query it as OMOP CDM 5.4 views. What is Avalon? Avalon is an end-to-end pipeline that turns Synthea-generated FHIR bundles into clean, documented, queryable tables in BigQuery , then la
Article URL: https://mistral.ai/news/leanstral Comments URL: https://news.ycombinator.com/item?id=47404796 Points: 286 # Comments: 53
Article URL: https://gluey.sh/ Comments URL: https://news.ycombinator.com/item?id=47404438 Points: 1 # Comments: 0
Serverless JARs and Databricks Connect for ScalaServerless JARs enable teams to build..
Stripe uses internal coding agents called 'Minions' to generate over 1,300 automated pull requests per week. The article likely describes the architecture and implementation of these agents.
With the launch of real-time mode (RTM) in Apache Spark 4.1, Structured Streaming..
ClickHouse-connect v0.12.0 introduces a new async-native Python client built using the half-sync/half-async pattern. Benchmarks show a 1.16x improvement in throughput and more stable tail latency under high concurrency.
We generated ~1,100 synthetic patients with Synthea, processed the FHIR R4 output through our normalization engine (Forge), and published it as a free public dataset on BigQuery Analytics Hub. 8 resource types: Patient, Encounter, Observation, Condition, Procedure, Immunization, MedicationRequest, D
We just open-sourced the core data sync engine behind Yeahchain. The problem we solved: traditional databases were hitting performance bottlenecks during high-frequency sync operations. For Yeahchain, we moved to a custom, lock-free architecture that maps shared memory regions directly to our proces
Article URL: https://simonwillison.net/guides/agentic-engineering-patterns/what-is-agentic-engineering/ Comments URL: https://news.ycombinator.com/item?id=47393908 Points: 127 # Comments: 76
https://github.com/seanwevans/lockstep I want to share my work-in-progress systems language with a v0.1.0 release of Lockstep. It is a data-oriented systems programming language designed for high-throughput, deterministic compute pipelines. I built Lockstep to bridge the gap between the productivity
This article reports on performance improvements in Redpanda using NVIDIA Vera, showing latency reductions and throughput gains compared to CPU models;
I spent a few years working at a company where all our microservices backed into MongoDB instances. We were constantly under top-down pressure to deliver fast, and because MongoDB is schemaless, it felt very easy to just add fields to our documents whenever we needed to expose data to another servic
Article URL: https://zzk273.github.io/LATENT/ Comments URL: https://news.ycombinator.com/item?id=47388273 Points: 143 # Comments: 30
Even seemingly simple engineering tasks — like updating an API — can become monumental undertakings when you’re dealing with millions of lines of code and thousands of engineers, especially if the changes are security-related. Meta uses AI codemods to automate security-related changes in their Andro
This ClickHouse blog post explains how to effectively query datetime columns, including examples for hourly bucketing and rush hour analysis using real taxi data.
This Dagster blog post introduces a course on AI-Driven Data Engineering, focusing on building production ELT pipelines with AI coding agents; staff+ engineers should be aware of emerging trends in AI-assisted data engineering, and this course might provide practical workflows for integrating AI int
Overview:Since we released the Databricks AI Security Framework (DASF) in 2024, the..
NBIM cut runtimes 30,40% in 3 months with the dbt Fusion engine and State-Aware Orchestration,without heavy optimization.
OpenAI details how ChatGPT is designed to resist prompt injection and social engineering by constraining risky actions and protecting sensitive data within agent workflows, offering insights into security measures.
OpenAI explains how they built an agent runtime using the Responses API, shell tool, and hosted containers to enable secure and scalable agents with access to files, tools, and state.
This post discusses the practical limits of DuckDB when running on commodity hardware. Understanding these limitations is crucial for optimizing performance and resource allocation in real-world deployments.
This post discusses a proposed patch set for Postgres 19 called pg_plan_advice, which would allow admins to control query planner behavior and improve plan stability;
Apple released the MacBook Neo today and there is no shortage of tech reviews explaining whether it's the right device for you if you are a student, a photographer or a writer. What they don't tell you is whether it fits into our Big Data on Your Laptop ethos. We wanted to answer this using a data-d
Learn how to build model-agnostic infrastructure for AI agents that works across multiple LLMs, enabling flexibility, scalability, and future-proof AI systems.
ETL, ELT, batch, CDC, reverse ETL,learn the key data movement patterns and when to use each.
Airbnb rolled out 20+ local payment methods in 360 days, and this article discusses the underlying technical architecture and engineering decisions.
This post demonstrates a lightweight metrics layer for ClickHouse using MooseStack, an open-source developer agent harness. It explains how to define metrics once and use them across different applications.
As an AI Architect, I spend my days designing AI systems and agents for others. I optimize workflows, fine-tune context windows, and architect serverless solutions to solve complex business problems.. View article
This article shares the technical details behind how Advanced Browsing Protection (ABP) in Messenger protects the privacy of the links clicked on within chats while still warning people about malicious links. It illuminates some of the engineering challenges and infrastructure required to implement
Replo uses ClickHouse to analyze over 100 billion events in real time. This enables fast analytics dashboards for over 4,000 Shopify merchants.
This article examines how the active role of AI agents, which act on data instead of merely reading it, necessitates a new form of digital twin. It explains why this shift requires integration within a live operational data infrastructure.
This article details the architectural design of a vector database built natively on object storage, focusing on how this approach enables efficient search capabilities. It explores the underlying principles and engineering trade-offs of such a design.
Netflix discusses how they are scaling global storytelling. The article dives into how they modernized localization analytics.
The article highlights the inefficiency of agentic systems that start from zero every run and proposes using cognitive memory to improve performance.
The article highlights an engineer at Cloudflare who rewrote most of Next.js in one week using AI agents; this example suggests a future where AI can rapidly disrupt existing software moats and business models, raising important questions about the evolving role of software engineers.
Explore Apache Iceberg v3 support in Snowflake public preview, including row lineage for CDC, variant data, and enhanced interoperability across open table formats.
The article introduces Iceberg output for Redpanda Connect, enabling users to land data directly into Apache Iceberg tables; it highlights advantages such as automated schema evolution and scalable routing, making it useful for those integrating streaming data with data lakes.
Discover how AI accelerates data lineage with automated docs, testing, and scalable governance.
Explore the decline of traditional RAG in the era of agentic AI, and how autonomous agents are reshaping retrieval, reasoning, and knowledge workflows.
We are entering the golden age of AI Coding. Every day, I see colleagues, both technical and non-technical, marveling at how agents are rewriting the rules of software construction. The.. View article
Netflix discusses optimizing recommendation systems. They use JDK’s Vector API to achieve this.
AI is reshaping data pipelines,driving real-time data, automation, and AI-assisted modeling at scale.
FFmpeg is a multi-tool for media processing, supporting a wide variety of audio and video codecs and container formats. It can also orchestrate complex chains of filters for media editing and manipulation. For the people who use our apps, FFmpeg plays an important role in ensuring that our videos lo
Meta recognizes the long-term benefits of jemalloc, a high-performance memory allocator, in its software infrastructure. Meta is renewing focus on jemalloc, aiming to reduce maintenance needs and modernize the codebase while continuing to evolve the allocator to adapt to the latest hardware and work
This article argues that traditional context engineering approaches alone are insufficient for optimal AI performance. It details how a live data architecture can effectively eliminate issues like context confusion, latency bottlenecks, and stale metadata, thereby powering production-ready AI agents
Context engineering alone won’t fix AI performance. The article explains how a live data architecture eliminates context confusion, latency bottlenecks, and stale metadata to power production-ready AI agents.
Stripe evaluated the ability of AI agents to create real Stripe integrations via benchmark. The benchmark tests how well AI agents can autonomously manage software engineering projects, despite their ability to solve scoped coding problems.
Unify metrics across BI and AI with Omni + the dbt Semantic Layer. Define once, query everywhere, eliminate semantic drift.
Apache Iceberg was built for batch analytics, but operational data changes continuously. This article explains how Materialize streams live, transactionally consistent data into Iceberg without the memory and latency costs of batching.
This Apache Doris blog post describes how Xiaomi built a unified data platform using Apache Doris, running 40+ clusters and serving 50 million queries per day across petabytes of data.
This article discusses the unique challenges of monitoring LLM agents due to their non-deterministic nature and infinite input possibilities; it proposes focusing on conversation quality and using production traces for continuous improvement, highlighting the shift from traditional software monitori
The article explains how Materialize uses self-correction to prevent output drift in materialized views, ensure consistency across upgrades, and enable in-place view replacement.
AI agents need more than engineered prompts; they need live, stateful context. This article explains how context engines deliver fresh, correct business objects in real time.
This article covers agent development with CockroachDB using the LangChain framework, highlighting the integration's support for building production-ready agentic AI applications.
Zscaler built an AI-powered, multi-agent PR review system that uses dbt’s structured context.
A surprising edge case involving row locks with joins in Postgres: non-null foreign keys and valid constraints do not guarantee an inner join will return a row under concurrent modifications. The post traces the exact sequence of operations that triggers the bug.
Discover why AI agents need ontology to structure knowledge, improve reasoning, enable semantic understanding, and make better autonomous decisions.
Netflix discusses DataJunction. The article explains how they use it as their answer to the missing piece of the modern data stack.
Meta is open-sourcing the initial version of RCCLX – an enhanced version of RCCL that we developed and tested on Meta’s internal workloads. RCCLX is fully integrated with Torchcomms and aims to empower researchers and developers to accelerate innovation, regardless of their chosen backend. Communica
This article delves into the architecture and implementation of StarRocks' high-performance vectorized engine, explaining how it enhances query processing speed and efficiency.
A Kubernetes beginner roadmap that goes through all k8s concepts with links to external documentation and exercises. TL;DR For the past few years, I’ve worked in startup environments where learning.. View article
Learn how to clone massive tables in ClickHouse instantly, without copying a single byte. Discover how immutable data parts and part-level copy-on-write make safe experimentation and migrations effortless.
The article provides reflections on four years at Materialize, sharing lessons on simplicity in system design, reusing core abstractions, balancing long-term vision with short-term execution, and understanding performance “speed limits” in distributed systems.
Netflix introduces MediaFM. The article explains how they use multimodal AI foundation for media understanding.
The post delves into the design and implementation of Agent Builder's memory system, discussing the prioritization of memory, technical architecture, and future enhancements; it offers valuable insight into building persistent memory systems for AI agents and their impact on performance.
The author investigates a map-reduce solution for querying 3 billion vectors, inspired by a discussion with Jeff Dean. The article delves into the implementation details of this solution, exploring the challenges and potential optimizations.
InMobi adopted ClickHouse and achieved 20x faster queries and 80% cost savings, building a fast, reliable, and cost-effective platform.
The Apache Iceberg community has finalized the File Format API, a significant architectural enhancement that enables pluggable, consistent, and engine-agnostic file formats within the Iceberg Java codebase.
This Spotify Engineering blog post discusses their multi-agent architecture for smarter advertising. The article likely details the challenges, solutions, and benefits of using a multi-agent approach to improve advertising effectiveness.
The article investigates a performance bottleneck where a customer query was significantly slower than expected. It details the process of using a profiler to identify problematic Rust code, tracing the issue to its assembly-level costs, and exploring the implications of zero-cost abstractions versu
The article details how to calculate the true cost of running Apache Flink, breaking down infrastructure, state management, and operational overhead for self-hosted and managed deployments.
Netflix discusses scaling LLM post-training. The article explains their approach and challenges.
Supabase provides a detailed account of the February 12 outage in us-east-2, explaining the root cause and the steps taken to prevent it from happening again. The article provides insight into the incident and the measures implemented to improve system reliability.
With intelligent orchestration and optimization, we achieved a 64% reduction in compute costs and simplified our job architecture.
This Netflix Tech Blog post discusses automating the migration of RDS Postgres to Aurora Postgres. The article likely details the challenges, solutions, and lessons learned during this process, offering insights for others undertaking similar migrations.
The article outlines a method for constructing a global distributed queue using a single JSON file stored on object storage. It describes the evolution of this system, starting with basic file usage and progressing to incorporate write batching, a stateless broker component, and high-availability.
This Apache Doris blog post covers how Apache Doris and Apache Paimon can be used to build a unified lakehouse for Web3 on-chain analytics, claiming 5x faster ETL than Spark and 2x faster data lake queries than Trino.
A practical guide to diagnosing GPU memory issues instead of randomly changing hyperparameters until something works Last week, I was building a reinforcement learning model for a customer using GRPO.. View article
Discover practical lessons from building OAuth flows with MCP apps, including OAuth 2.0 patterns, security issues, and implementation tips.
This Netflix Tech Blog post covers high-throughput graph abstraction. The article likely describes the architecture, implementation, and performance considerations of their graph abstraction system, offering practical insights for building similar systems.
This article shares details of the role backend aggregation (BAG) plays in building Meta’s gigawatt-scale AI clusters like Prometheus. BAG allows Meta to seamlessly connect thousands of GPUs across multiple data centers and regions. Their BAG implementation is connecting two different network fabric
This Netflix Tech Blog post details how Netflix validates catalog metadata using a 'Data Canary' system. The article likely explains the architecture, implementation, and benefits of this system for ensuring data quality and reliability.
This Apache Doris blog post explains how to build a real-time Web3 analytics platform using Apache Flink and Apache Doris, enabling sub-second queries on billions of blockchain transactions.
This article discusses how context graphs can improve AI agent performance, emphasizing the shift from simple state management to incorporating semantic understanding of the data; this is.
This Netflix Tech Blog post discusses 'Data Bridge', a system Netflix uses to simplify data movement. The article likely explains the architecture, implementation, and benefits of this system for improving data pipeline efficiency and reducing complexity.
This post explores how an individual replaced a paid SaaS subscription with LLM-generated code in just 20 minutes; this highlights the potential for LLMs to disrupt simple SaaS business models, especially for products that are not actively maintained.
This Netflix Tech Blog post covers the AI evolution of graph search at Netflix. The article likely describes how they're using AI to improve graph search capabilities, offering insights into building intelligent search systems.
The article shares lessons learned from observing billions of agentic workflows, focusing on the challenges of moving from a working demo to a production system.
I think it is worth starting this intro by talking a little bit about the established format for columnar data. Parquet has done some amazing things for analytics. If you go back to the times where CSV was the better alternative, then you know how important Parquet is. However, even if the specific
The article argues that human-in-the-loop is a missing layer in agentic systems.
This StarRocks blog post dives into the details of join optimization within the StarRocks database, explaining why joins can perform faster than expected. The author is a StarRocks committer and engineer at Celerdata.
This article introduces the latest version of an Approximate Nearest Neighbor (ANN) system, highlighting its capability to handle over 100 billion vectors within a single search index. It reports achieving a p99 query latency of 200ms at 1,000 queries per second (QPS) while maintaining 92% recall.
Building Semantic Tool Selection with Multi-Component Embeddings Last quarter, our enterprise AI platform hit a wall. We had built an impressive suite of 70+ automated tools covering everything from database.. View article
Apache Doris now supports native hybrid search for AI workloads. The new functionality allows vector search, full-text search, and structured analytics within a single SQL engine, enabling AI-powered applications to leverage a unified data platform.
Deep technical guide to dbt-expectations covering regex validation, freshness/SLA checks, completeness validation within time windows, JSON schema validation, statistical distribution checks, and cross-column logic. Shows integration with production monitoring.
Examines how the EU AI Act, Cyber Resilience Act, and Data Act turn messy data from a performance tax into a legal liability. Covers the August 2026 deadline for High-Risk AI system compliance and argues governance must shift from reactive cleanup to embedded-by-design architecture.
The article describes the redesign of an inverted index structure, detailing the adoption of fixed-sized posting blocks within a key-value store built on object storage. This architectural change resulted in a tenfold reduction in index size and a dramatic increase in system throughput.
The velocity of Generative AI has been nothing short of relentless. In the span of just 24 months, the industry has shifted paradigms three times. We started with the raw.. View article
This article presents a critique of the Iceberg REST catalog, arguing that a semantically correct API can become operationally unreliable at scale;
This Spotify Engineering blog post explains the technical and practical rationale for using separate tech stacks for personalization and experimentation. The article likely details the benefits of this separation, such as improved agility and scalability.
This article presents an analysis of how BM25 query latencies vary with document count and the top_k parameter. It explores surprising scaling characteristics, noting that longer queries may scale less efficiently and that the presence of essential terms can impact performance in unexpected ways.
This post outlines building a real-time lakehouse architecture using Redpanda's Iceberg Topics and Databricks Unity Catalog for analytics-ready tables, eliminating the need for batch processing and orchestration, which is of interest to practitioners.
Weaviate 1.35 introduces Object Time-to-Live (TTL), zstd compression support, flat index RQ quantization, multimodal support with Weaviate Embeddings, and runtime configurable OIDC certificates.
Apache Doris introduces the VARIANT data type for high-performance JSON analytics. Optimizations such as dynamic subcolumns, sparse columns, schema templates, lazy materialization, and path-based indexing allow it to outperform PostgreSQL and MongoDB in JSON handling.
This StarRocks blog post provides 10 tips for optimizing OLAP query performance. The author is a StarRocks TSC member and query engine team lead at CelerData.
Learn how to build secure, observable agentic AI on Redpanda using Connect for AI tools and data access, plus broker audit logs to capture every agent action.
ByteDance uses Apache Doris 4.0 to solve billion-scale vector search problems. The company leverages Doris's hybrid search capabilities to build a system that balances accuracy, low latency, and cost-efficiency when handling over 1 billion vectors.
In this post, we describe the current patterns for interacting with Iceberg Catalogs, and pose the question: could it be done from a browser? After elaborating on the DuckDB ecosystem changes required to unlock this capability, we demonstrate our approach to interacting with an Iceberg REST Catalog.
The article identifies architecture as the key challenge in building production multi-agent systems, based on insights from 1.7 billion workflows.
Browser-native SQL workbench built on DuckDB WASM. Query CSV, Parquet, Excel, JSON files locally with zero install. Supports AI SQL assistant, BigQuery connector, and shareable URLs. Your data never leaves your machine.
This article proposes a model extending generic durable functions into three forms: stateless functions, stateful function objects, and linear function chains. It aims to standardize terminology in durable execution engines by linking concepts like 'workflows' and 'activities' to underlying executio
The article describes how text search performance has been improved by up to 20x through the adoption of a vectorized variant of the block-max MAXSCORE algorithm, a technique also employed by Apache Lucene. This enhancement is particularly relevant for handling long queries generated by large langua
This article discusses context engineering, focusing on how AI agents manage LLM memory by selecting, retrieving, and organizing context from short-term and long-term memory. Context engineering is important for improving the reliability of AI agents in production.
This post delves into the architecture of durable function trees, exploring their integration within larger systems and the advantages they offer for durable execution.
This article explores constructing workflows using durable function calls arranged in trees, built on durable promises and continuations.
This article announces a substantial upgrade to a full-text search engine, promising up to a 20x improvement in search performance. The upgrade reflects significant enhancements made to the underlying search architecture.
Over the past several months, the DuckDB Labs team has been hard at work on the DuckDB-Iceberg extension, with full read support and initial write support released in v1.4.0. Today, we are happy to announce delete and update support for Iceberg v2 tables is available in v1.4.2! The Iceberg open tabl
This article explains the concept of determinism within durable execution frameworks, focusing on identifying code sections that must be deterministic.
This article explains how to leverage the Dify and Weaviate integration for building Retrieval Augmented Generation (RAG) applications. This integration can be valuable for enhancing LLM applications with external knowledge.
Technical overview of Apache Polaris as the emerging open catalog standard for Iceberg. Covers multi-engine interoperability (Spark, Flink, Trino, StarRocks), built-in RBAC with table-level security, short-lived credential vending via cloud provider integrations, and Snowflake's managed Polaris offe
This article introduces Qbeast, an OTree spatial index designed for data lakehouses, and discusses its integration with Apache Iceberg and Delta Lake.
Weaviate 1.34 introduces flat index support with RQ quantization, server-side batching improvements, new client libraries, and Contextual AI integration. These features offer potential performance and functionality improvements for the vector database.
Apache Doris achieves top performance in the JSONBench benchmark, particularly in cold query performance and data quality. The benchmark measures query performance and data handling capabilities when processing JSON data.
This article discusses how stream and batch analytics can be built on Apache Iceberg, highlighting potential conflicts due to their differing requirements.
This article explores the architectural considerations and engineering approaches necessary for building vector storage systems capable of scaling to billions of vectors. It specifically addresses these challenges within the context of Retrieval Augmented Generation (RAG) applications.
Apache Doris achieves 70% better price-performance on ARM-based AWS Graviton instances compared to x86. The benchmark results, gathered from standard OLAP tests such as ClickBench, SSB, TPC-H, and TPC-DS, demonstrate the efficiency of Doris on ARM architecture.
This article highlights the emerging trend of developers utilizing multiple AI agents in parallel to generate code. It explores the potential benefits and challenges of this approach to programming.
The article describes the process of building a new vector database, detailing the architectural choices, design philosophy, and implementation challenges encountered. It outlines how specific technical hurdles were addressed during its development.
This article examines the Kafka community's efforts to lower replication costs by discussing KIP-1150, KIP-1176, and KIP-1183.
The article announces the initial major release of `airflowctl` 0.1.0, a new secure and API-driven command-line interface for Apache Airflow. This CLI is designed to align with modern API communication and auditability standards.
Comprehensive landscape of open-source data quality tools including Soda Core, Elementary Data, dbt Tests, and DataKitchen TestGen. Explores how the community is democratizing observability capabilities previously locked behind expensive platforms, and how AI is being used to automate test generatio
This article examines query performance optimization techniques in open table formats such as Apache Iceberg, highlighting methods beyond standard indexing.
Apache Doris is shown to be significantly faster than ClickHouse in real-time updates, according to benchmark results. Using ClickBench and SSB (Star Schema Benchmark), Apache Doris outperforms ClickHouse by 18-34x in SSB and 2.5-4.6x in ClickBench.
Argues that most organizations place data contracts in the wrong part of the lifecycle, causing enforcement gaps. Makes the case for contracts closer to the producer, not the consumer, with practical guidance on where they should sit architecturally.
The article announces the release of Apache Airflow 3.1.0, an update that integrates human decision-making into automated processes. It also introduces comprehensive internationalization support and substantial developer experience enhancements.
Learn how Search Mode compares against Hybrid Search on the BEIR, LoTTe, BRIGHT, EnronQA, and WixQA Information Retrieval benchmarks.
This article describes training an LLM that can converse in English and item IDs, making recommendations without retrieval or tools.
Apache Doris utilizes various data pruning techniques to optimize query performance by skipping unnecessary data processing. This article dives into the implementation and strategies behind these data pruning techniques within the Doris architecture.
Apache Doris demonstrates superior performance over ClickHouse in various benchmarks including CoffeeBench, TPC-H, and TPC-DS. The benchmarks show that Doris consistently outperforms ClickHouse, showcasing its efficiency and speed in OLAP workloads.
Production-oriented guide showing how to capture column-level lineage in Microsoft Fabric Spark (which ships with OpenLineage pre-installed). Describes a Spark Plugin architecture where a REST API collects lineage events from an OpenLineage Listener, buffering them into Delta Tables for queryable li
Learn how chunking strategies improve LLM RAG pipelines, retrieval quality, and agent memory performance across production AI systems.
This post delves into the internal workings of Apache Fluss, offering a detailed exploration for those interested in data system internals.
Get spun around by our new vector quantization algorithm that utilizes the power of random rotations to improve the speed-quality tradeoff of vector search with Weaviate.
This article introduces a conceptual model for storage unification, designed to present diverse storage systems and formats as a unified resource.
Compares next-generation Iceberg catalogs: Nessie (Git-style branching for data), Apache Polaris, Apache Gravitino, Lakekeeper, and Unity Catalog. Explains how these move beyond simple table-name resolution to provide version control, federated views, fine-grained policies, and multi-engine freedom.
Side-by-side technical comparison of Great Expectations, Soda Core, dbt tests, and Deequ across expressiveness, scalability, integration patterns, and ease of adoption. Provides a decision framework for which tool fits which use case, and discusses layering multiple tools across pipeline stages.
Modal claims that open models can transcribe speech 100 times faster and 100 times cheaper than previous methods.
Builds a complete order processing pipeline with Debezium CDC, Apache Flink transformations, and OpenLineage/Marquez for lineage tracking. Demonstrates how lineage metadata enables root cause analysis when pipeline failures occur, showing practical troubleshooting patterns with end-to-end visibility
Features Russell Spitzer (Apache Iceberg/Polaris PMC) discussing the distinction between business catalogs (discovery/listing) and system catalogs (governing access by understanding table layout). Covers how Polaris vends short-lived credentials scoped to exact table directories.
Constella built a cross-platform thinking tool using Weaviate, RAG, and a multi-tenant architecture. The post details how they implemented vector search and syncing across devices.
This article covers evaluation metrics, how to build eval datasets, evaluation methodology, and a review of several benchmarks for long-context question and answer systems.
Production case study from Whatnot (live shopping marketplace) on combining data contracts with Monte Carlo observability. Their stack uses Snowflake, dbt, and Dagster. Shows how enforcing contracts while layering automated observability kept data incidents flat despite exponential data growth.
This article delves into the difficulties of operating Apache Flink in production environments. It explores the reasons why Flink is considered challenging and provides insights into how to address these operational complexities.
Technical walkthrough of Debezium's built-in OpenLineage integration for automatic CDC lineage tracking. Explains how Debezium Server emits OpenLineage events natively using the Java SDK, modeling run/job/dataset entities without manual instrumentation, with Marquez as a lineage backend.
Covers Unity Catalog announcements: Iceberg catalog federation for governing tables in AWS Glue/Hive/Snowflake without copying data, Unity Catalog Metrics as first-class governed assets, column-level permissions for PII, and the new Discover experience for certified data products with AI-driven reco
Weaviate version 1.31 introduces the MUVERA encoding algorithm for multi-vector embeddings. The post explains the algorithm's details, including its functionality and use cases.
This article discusses how Recsys and search are converging with LLMs via semantic IDs, data augmentation, and unified foundation models.
DuckLake stores data lake metadata in a SQL database instead of files. 22-table schema replaces manifest files, enabling instant snapshot queries and ACID transactions without file listing overhead.
This article details automating agentic workflows using Amazon Q CLI, Anthropic MCP, and tmux to build news agents for daily news recaps.
Explains the columnLineage dataset facet introduced in OpenLineage 0.9.0 for Spark integration. Covers how column-level lineage tracks which input fields produce each output field, its applications for GDPR/HIPAA/CCPA compliance, and the roadmap for extending support beyond Spark.
The article announces the general availability of Apache Airflow 3.0, marking the project's largest release in its history. This milestone release culminates four years of development and introduces substantial changes to the Airflow platform.
This article provides guidance on testing custom Flink jobs on Decodable, focusing on modular implementations to improve testability when dealing with external service dependencies. It addresses a common challenge in Flink development and offers practical solutions.
This article demonstrates integrating Neo4j with Qdrant to enhance RAG pipelines by enabling external vector searches; it guides users through a local setup with preloaded data, illustrating the practical aspects of this integration.
The article explains how to build knowledge graph agents using LlamaIndex workflows, offering a blueprint for constructing Text2Cypher agentic interfaces; this integration provides practical insights into developing agentic data pipelines.
This article identifies common pitfalls encountered when building generative AI applications and provides examples.
This Neo4j Developer Blog post explains how to use Anthropic's Model Context Protocol (MCP) to give LLMs like Claude access to knowledge graphs in Neo4j.
Anthropic's guide to building reliable AI agents: tool use patterns, prompt chaining, evaluation frameworks, error recovery, and when NOT to use agents.
Integrate Neo4j knowledge graphs with LangChain for powerful GraphRAG applications that deliver deeper, more insightful answers.
Explore how GraphRAG can be used to streamline the process of ingesting commercial contract data and building a Q&A Agent.
MCP standardizes how AI models connect to data sources and tools. Client-server architecture with typed resources, tool definitions, and prompts that any LLM application can implement.
Dive into the impact of fine-tuning models for the Text2Cypher task of transforming natural language questions to Cypher queries.
Learn how to build a support agent that relies on information from Stack Overflow using the GenAI Stack – Neo4j, LangChain & Ollama in Docker.
Explore performance benchmarks of LLM models on Neo4j Text2Cypher (2024) Dataset, comparing foundational vs. fine-tuned models for Cypher query translation.
The Text2CypherRetriever allows users to retrieve data from Neo4j using natural language, simplifying query generation for GenAI applications.
DuckDB ships a built-in web UI for interactive SQL exploration, schema browsing, and result visualization -- no install needed beyond the CLI.
The post describes how to monitor Neo4j in a clustered environment using tools like Dynatrace and Kibana, along with best practices.
This technical blog post explores the importance of Change Data Capture (CDC) for developers. It covers the fundamentals of CDC, its common use cases, and the advantages of log-based CDC compared to other approaches. Understand how CDC can improve operational performance, enable real-time analytics,
The new GraphAcademy course teaches how to convert unstructured data into graphs using GenAI, LLMs, and Python.
Neo4j’s graph database enables real-time analysis to uncover hidden fraud rings and protect financial assets, aiding in bank fraud detection.
The post details how to turn CSV files into graph models using LLMs, simplifying data relationships and enhancing insights.
Learn to build accurate, explainable recommendation systems with minimal code using Neo4j graph database and Keymaker framework.
Learn how to build a GraphRAG agent using Neo4j and Milvus, combining graph and vector search for enhanced retrieval, better context, and accurate answers.
Learn how to overcome the challenges of structured data operations in text embeddings in RAG applications using knowledge graphs.
Enhance GraphRAG applications by combining hybrid search and graph traversal with Neo4j’s HybridCypherRetriever, improving retrieval for complex queries.
Learn how Debezium, the de-facto standard for open-source change data capture (CDC), has evolved to support deployments without the need for Kafka-related infrastructure.
How enterprises combine knowledge graphs with LLMs: grounding responses in structured facts, reducing hallucinations, enabling explainable AI, and the architectural patterns for graph-augmented generation.
Explore advanced GraphRAG retrieval patterns and how graph structures enhance RAG systems. Learn actionable strategies to implement and optimize GraphRAG.
Prefect 3.0 drops DAGs entirely: Python-native flows with dynamic task creation, automatic retries, event-driven triggers, and a hosted platform that eliminates scheduler management.
Recommend movies to users based on their reading histories and ratings. Learn the setup of Neo4j, mapping data into Java with Neo4j Object Graph Mapper (Neo4j-OGM), and crafting Cypher queries for recommendations.
LLMs generating SQL without a semantic layer produce inconsistent, wrong metrics. How the dbt Semantic Layer provides guardrails: metric definitions, entity relationships, and governed access for AI agents.
The article details the application of Direct Preference Optimization (DPO) techniques utilizing synthetic data. It explores how these methods are implemented and leveraged within the Anyscale platform for machine learning model refinement.
Architecture-level comparison: Polars' Rust-based columnar engine with lazy evaluation, query optimization, and Apache Arrow memory vs Pandas' eager NumPy-backed row operations. Benchmarks on real workloads.
A comprehensive guide on troubleshooting and configuring Flink SQL to write to Delta Lake on S3 or MinIO.
Inside Snowflake's Cascades-style query optimizer: join reordering, pruning with micro-partition statistics, adaptive execution, and how they test optimizer correctness at scale.
Ontologies provide the structured backbone that LLMs lack: taxonomies, controlled vocabularies, entity disambiguation, and how combining ontological reasoning with neural approaches produces more reliable AI systems.
DataFusion as a modular query engine: how it powers InfluxDB 3.0, Comet Spark accelerator, and Ballista distributed queries. Extensible optimizer, custom table providers, and user-defined functions in Rust.
This article explores declarative resource management in Decodable, highlighting its benefits for SDLC best practices, resource management, environment migration, and resource cleanup.
The asset-centric paradigm shift: why defining what data should exist (Dagster assets) is better than defining how to compute it (Airflow tasks). Software-defined assets, IO managers, and testability.
Analysis of how Iceberg's catalog-agnostic design, hidden partitioning, and multi-engine support gave it an architectural advantage over Delta Lake and Hudi.
Practical data governance: automated PII detection, column-level lineage, data contracts between teams, freshness SLAs, and how to implement governance incrementally without blocking teams.
Building real-time ML feature pipelines with Flink: window aggregations, CDC ingestion, point-in-time joins, and integration with Feast and Tecton feature stores.
How DuckDB's community extension system works: writing C++ extensions, the extension repository, signed distribution, and examples of spatial, httpfs, and Iceberg extensions.
Microsoft's GraphRAG approach: automatically building knowledge graphs from document corpora, community detection for topic summarization, and how graph-based retrieval answers global questions that vector search cannot.
This post describes a method for building an LLM router that dynamically selects the optimal LLM for a given request based on configurable criteria. It covers techniques for evaluating LLM performance, implementing routing logic, and optimizing for cost-effectiveness.
Using LLMs to extract entities and relationships from documents, resolve coreferences, and populate a Neo4j knowledge graph. Includes schema design, prompt engineering for extraction, and evaluation metrics.
Snowflake Dynamic Tables: define a pipeline as a SQL query and let Snowflake handle scheduling, incremental refresh, and dependency management. Replaces streams + tasks for most use cases.
Stripe's ledger system for financial data: immutable event log, double-entry accounting in the data warehouse, reconciliation pipelines, and how they ensure every cent is accounted for.
DuckDB's relational API as a replacement for Pandas in ETL pipelines, with benchmarks showing 10-100x performance improvements on larger-than-memory datasets.
The article explores the architecture of multitenant serving applications built on Ray, emphasizing the use of containerized runtime environments. It likely covers strategies for isolating different workloads, managing dependencies, and ensuring efficient resource utilization in production ML deploy
Databricks open-sources Unity Catalog, providing a universal governance layer across Delta, Iceberg, and Hudi tables with fine-grained access control and lineage tracking.
Honest comparison of ClickHouse and Snowflake architectures for real-time analytics workloads, covering query latency, ingestion throughput, cost models, and operational complexity.
Hard-won lessons from practitioners: prompt engineering diminishing returns, when to fine-tune vs RAG, evaluation beyond vibes, cost optimization, and the reliability gap between demo and production.
Airbnb's Midas data quality framework: automated anomaly detection, lineage-based impact analysis, SLA tracking, and self-healing pipelines at petabyte scale.
How LinkedIn processes 7 trillion events per day: Kafka for event transport, Samza for stream processing, Venice for derived data serving, and Brooklin for cross-DC replication.
Asset-centric vs task-centric orchestration: how Dagster's software-defined assets, type system, and built-in IO managers compare to Airflow's DAG paradigm.
When Postgres analytics hits a wall: column compression, vectorized execution, and approximate query processing in ClickHouse vs row-oriented scans in Postgres. Migration patterns and hybrid architectures.
Why dimensional modeling still matters even though the ELT era made star schemas seem obsolete. The semantic layer as the modern replacement for physical dimension tables.
Why feature stores exist: the training-serving skew problem, online vs offline stores, feature computation patterns, and how Feast, Tecton, and Hopsworks compare architecturally.
Netflix's migration from Hive to Iceberg at exabyte scale, including incremental processing patterns with Maestro orchestrator and Spark.
Why vector similarity alone fails for complex reasoning. Using Neo4j knowledge graphs alongside embeddings: entity extraction, relationship mapping, graph traversal for multi-hop queries, and hybrid retrieval.
Figma's horizontal sharding journey: from a single Postgres instance to 100+ shards using PgBouncer, application-level routing, and their custom migration tooling for zero-downtime resharding.
Why LLM-generated SQL fails in production: schema ambiguity, implicit business logic, multi-table joins, aggregate semantics, and why a semantic layer is the real solution instead of better prompting.
How data contracts formalize the interface between producers and consumers, with practical schema enforcement patterns using protobuf, JSON Schema, and dbt tests.
Advanced RAG patterns: multi-query retrieval, recursive summarization, parent-child chunk linking, self-RAG with reflection, and corrective RAG that verifies its own retrievals.
How Cube's semantic layer sits between databases and consumers: pre-aggregations, access control, caching, and serving consistent metrics to dashboards, notebooks, and LLMs via API.
End-to-end guide for building production RAG systems: chunking strategies, embedding model selection, retrieval metrics (MRR, NDCG), reranking, and hallucination detection.
Practical guide to running Kafka with KRaft consensus, covering migration from ZooKeeper, operational considerations, and performance characteristics.
How the dbt Semantic Layer works: MetricFlow engine, semantic models, dimension/measure definitions, and querying metrics from any BI tool via the JDBC/GraphQL API.
The original modern data stack (Fivetran + Snowflake + dbt + Looker) matured into a commodity. What comes next: embedded analytics, semantic layers, and AI-native data tools.
Production-grade vector search with pgvector: HNSW vs IVFFlat index tradeoffs, optimal dimensionality, bulk loading strategies, and benchmarks against dedicated vector databases.
Spark Connect introduces a thin client protocol that separates Spark applications from cluster infrastructure, enabling remote execution without shipping JARs or managing classpaths.
Production LLM serving: continuous batching vs static batching, PagedAttention (vLLM), speculative decoding, KV cache optimization, and how to maximize GPU utilization.
The semantic web stack (RDF, OWL, SPARQL) is quietly powering enterprise knowledge management. How knowledge graphs, linked data, and ontologies are being integrated with LLMs and modern data architectures.
Spotify's migration from proprietary data warehouse to an open lakehouse stack: partition pruning strategies, compaction scheduling, and how they cut compute costs by 40%.
Using DuckDB as a command-line JSON processor, replacing jq for complex data transformations with SQL syntax.
Production Airflow patterns: KubernetesExecutor vs CeleryExecutor, DAG serialization, connection management, XCom anti-patterns, and monitoring with StatsD and Prometheus.
Flink's roadmap toward true batch-stream unification, materialized tables, and the new ProcessFunction API for stateful event processing.
Deep dive into Postgres internals: shared_buffers vs OS cache, parallel query tuning, JIT compilation tradeoffs, connection pooling with PgBouncer, and VACUUM strategies for write-heavy workloads.
The case for using Postgres as your only database: JSONB for documents, pg_cron for scheduling, pgvector for embeddings, logical replication for CDC, and extensions for everything else.
What a semantic layer actually is beyond marketing: universal metric definitions, entity relationships, access policies, and why it matters more in the age of LLM-generated SQL.
Discord's migration from Cassandra to ScyllaDB for their message store: hot partition detection, consistent hashing, compaction tuning, and achieving P99 reads under 1ms at 2T messages.
Side-by-side comparison of the three major open table formats covering ACID semantics, schema evolution, time travel, compaction, and ecosystem support.
How dbt Mesh enables cross-project dependencies, model contracts, and versioning for organizations running hundreds of dbt projects across teams.
How ClickHouse's MergeTree engine works: LSM-tree-inspired sorted parts, sparse primary index, data skipping indexes, background merges, and why it achieves sub-second queries on billions of rows.
Practical fine-tuning guide using TRL, QLoRA, and Flash Attention 2. Covers dataset preparation, hyperparameter selection, evaluation, and deployment with real cost breakdowns.
Photon is a C++ vectorized execution engine that replaces Spark's JVM-based Catalyst for scan-heavy workloads, achieving 3-8x speedups through SIMD, memory-mapped I/O, and adaptive execution.
How the analytics engineer role evolved from a dbt power user to a critical bridge between data engineering and business intelligence, with practical career guidance.
WarpStream's architecture: a Kafka-compatible broker that writes directly to S3 instead of local disks. No inter-broker replication, no partition reassignment, and 80% cheaper than self-hosted Kafka.
Uber's integrated caching layer that combines Docstore with an in-process cache, handling 40M reads/sec with sub-millisecond P99 latency.
Companion post to the O'Reilly book covering Iceberg's hidden partitioning, schema evolution, time travel, and compaction strategies for production lakehouses.
Practical guide to Data Vault 2.0: hub-link-satellite patterns, hash keys for parallelism, point-in-time tables, and when Data Vault makes sense vs One Big Table or dimensional modeling.
How Arrow's in-memory columnar format enables zero-copy data exchange between Spark, DuckDB, Pandas, Polars, and databases via ADBC and Flight SQL.
The mechanics behind Kafka's exactly-once guarantees: idempotent producers, transactional messaging, consumer group coordination, and the performance cost of EOS vs at-least-once.
Martin Kleppmann's reflections on how the landscape has changed since DDIA: new consensus protocols, CRDTs in production, the shift to event streaming, and what he'd write differently today.
DuckDB's multi-database support: attach Postgres, MySQL, and SQLite databases alongside local files, and query across them with standard SQL joins.
Reference architecture for LLM applications covering RAG pipelines, embedding models, vector databases, orchestration frameworks, and evaluation patterns.
Anyscale's blog post describes a technique to reduce the cost of embedding computations in retrieval-augmented generation (RAG) pipelines by 10x using Anyscale and Pinecone. This approach is .
Quantitative analysis of GPU options for LLM inference and fine-tuning: comparing H100 vs A100 vs consumer GPUs, quantization tradeoffs (GPTQ, AWQ, GGUF), and cost-per-token calculations.
DuckDB's SQL dialect extensions that make queries more readable: GROUP BY ALL, SELECT * EXCLUDE, implicit column aliases, and string slicing.
Deep comparison of Kafka and Pulsar replication models: ISR vs BookKeeper quorum writes, tail latency characteristics, data loss scenarios, and how each handles broker failures.
Visual walkthrough of how Stable Diffusion works: the latent space, the denoising U-Net, CLIP text encoder, classifier-free guidance, and how LoRA fine-tuning adapts the model.
Internal architecture of Snowflake's multi-cluster shared data architecture, covering storage layer, virtual warehouses, metadata store, and query optimization.
The MoE architecture behind Mixtral and Switch Transformer: expert routing, load balancing, training instability, and why sparse models achieve better performance per FLOP than dense models.
The definitive visual explanation of the Transformer architecture: self-attention, multi-head attention, positional encoding, and how information flows through encoder-decoder layers.
Whatnot went from no modern data stack to processing tens of millions of events across hundreds of event types each day. Zack Klein explains how Whatnot leverages data contracts and data observability to achieve high quality data at scale for stakeholders, focusing on a small team's approach to data
Solving the viral 1 Billion Row Challenge using DuckDB SQL instead of Java -- demonstrating that a single SQL query on a laptop can process 1B rows in under 4 seconds.
Jay Kreps' foundational essay on the unified log abstraction, how it connects databases, distributed systems, and real-time data -- the intellectual basis for Kafka.
Vishnu Ram from Credit Karma discusses data reliability for LLMs and offers best practices related to data observability in AI.
Ed Presz from Pie Insurance explains how his team built an incident detection and notification workflow to drive ownership of data quality for business stakeholders.
The article emphasizes that for GenAI, data observability must prioritize resolution, pipeline efficiency, and streaming/vector infrastructures.
This article provides a case study focused on LLM-based summarization, comparing the quality of summaries generated by humans, Llama 2 70b, and GPT-4. It evaluates their respective performances in summarization tasks.
Reference, context, and preference-based metrics, self-consistency, and catching hallucinations.
The article presents findings indicating that Llama 2 achieves comparable factual accuracy to GPT-4 in summarization tasks. It further highlights that Llama 2 is approximately 30 times more cost-effective for these operations.
Evals, RAG, fine-tuning, caching, guardrails, defensive UX, and collecting user feedback.
Discover PyTorch's journey in an episode with Soumith Chintala, its Co-Creator and Meta's VP/Fellow. Learn about TensorFlow's impact, community-guided innovation, and the open vs. closed-source debate.
This article explores the purpose and impact of fine-tuning in large language models. It posits that fine-tuning primarily influences the stylistic and structural "form" of an LLM's output rather than imparting new factual information.
Describes Aviary, an open-source solution designed for multi-LLM serving. The post introduces the project's capabilities and its role in managing diverse large language models within a single serving infrastructure.
Presents essential numerical data that LLM developers should be aware of, covering performance metrics, operational costs, or scaling factors relevant to deploying and managing large language models. The article provides practical insights for optimizing LLM systems.
In this article, we explore how to utilize OpenAI's ChatGPT and LangChain to build a Question-Answering bot for Weights & Biases' podcast series, Gradient Dissent.
Explores methods for fine-tuning and serving LLMs efficiently and cost-effectively, leveraging technologies such as Ray, DeepSpeed, and HuggingFace. The article is the fourth in a series, building upon previous discussions of Ray's capabilities for generative AI infrastructure and performance optimi
This article focuses on data contracts for data warehouses, emphasizing programmatic accountability in batch data processing. It outlines the importance of defining and enforcing data contracts to improve data quality and reliability.
This Anyscale blog post dives into data ingest in a third-generation ML architecture, specifically using Ray Data. It provides code samples to illustrate how distributed libraries can improve performance by exploiting distributed memory bandwidth.
Discusses the evolution of production machine learning architectures, categorizing them into generations. The article describes the shift from fixed-function pipelines to programmable pipelines, and then speculates on the characteristics of the emerging third generation of ML architectures.
XGBoost-Ray is a new backend for distributed XGBoost training that supports multi-node and multi-GPU setups. It includes distributed data loading, fault tolerance with elastic training, and integrates with the Ray Tune hyperparameter optimization framework.
Ray 1.2.0 introduces a new library of collective communication primitives designed to streamline information exchange across numerous distributed processes. These primitives aim to simplify distributed operations within Ray programs and provide substantial speedups, potentially by an order of magnit