Data pipelines: AI agents, skills & prompts

Guide for contributing to the Airflow Java SDK (AIP-108). Use this skill

Guide for implementing a brand-new language SDK for Airflow (AIP-108). Use

Draft Apache Airflow commit messages, PR titles and descriptions, or GitHub comments and reviews. Also use for release-note decisions when preparing a contribution.

Databricks specifics for Atlas: the extended binary and login, connection URLs and scope (catalog vs schema), the dev catalog (no Docker), Delta table properties, managed and external tables,…

Build production Apache Airflow DAGs with best practices for operators, sensors, testing, and deployment. Use when creating data pipelines, orchestrating workflows, or scheduling batch jobs.

Map dbt `schema.yml` / `properties.yml` models and sources into ktx semantic-layer overlays and column notes. Covers `sources:` vs `models:`, column `data_tests` (not_null, unique, accepted_values,…

星火燎原：从零起步、资源极少时，先评估客观条件，选定一个能站稳、能复用、能扩展的"根据地"（最小切入点），扎根后再逐步扩张，拒绝四处出击的流寇主义。当启动新项目、做 MVP / pilot、面对宏大目标不知从何下手、或做了很多零散事却没有积累时触发；项目已有稳固基础、只是修改现有系统时不触发。

Execute arbitrary Python or PySpark code on Fabric Spark compute without creating a notebook artifact; ephemeral Livy sessions with full Delta table access. Automatically invoke when the user asks…

Use for any task involving Dagster, the dg CLI, assets, materialization, components, data tools or data

Generates a Mermaid flowchart diagram of dbt model lineage using MCP tools, manifest.json, or direct code parsing as fallbacks. Use when visualizing dbt model lineage and dependencies as a Mermaid…

Use when a user needs help triaging dbt-core to dbt v2 migration errors. Runs dbt-autofix first, then classifies remaining errors into actionable categories (auto-fixable, guided fixes, needs input,…

Use when migrating a dbt project from one data platform or data warehouse to another (e.g., Snowflake to Databricks, Databricks to Snowflake) using dbt Fusion's real-time compilation to identify and…

Automatically route Snowflake prompts from Claude Code to Cortex Code for execution. Provides slash commands for code review and task delegation, plus skills for routing, run, and setup.

Builds human-in-the-loop (HITL) Airflow workflows - approval gates, form input, and human-driven branching. Use when a DAG needs a human in the loop - an approval or reject step, sign-off before a…

Builds Airflow 3.1+ plugins that embed FastAPI apps, custom UI pages, React components, middleware, macros, and operator links directly into the Airflow UI. Use when building anything custom inside…

Persists task and asset state across retries and DAG runs using Airflow 3.3's AIP-103 key/value stores (`task_state_store`, `asset_state_store`) and the crash-safe `ResumableJobMixin`. Use when the…

Connect, query, and generate data insights for BigQuery datasets and data.

Leverages BigQuery's built-in machine learning and GenAI capabilities

Generates Python code using BigQuery DataFrames (BigFrames). Use by default for any Python data task involving BigQuery, including data processing, analysis, and machine learning. Don't use for…

Discovers and inspects BigQuery Data Transfer Service (DTS) configurations.

Drive the Oracle AI Data Platform (AIDP) Databricks Migration Toolkit in natural language. Plans and executes automated Databricks → AIDP migrations of notebooks, jobs, schedules, and catalog DDL —…

Oracle AI Data Platform Workbench Spark connectors for Claude Code. 18 connector skills covering every data source workbench customers commonly need: Oracle Autonomous DB family (ALH/ADW/ATP) via…

Builds robust data pipelines with schema evolution, data quality checks, incremental loading, and fault-tolerant processing

Apache Airflow 3.0 SME for DAG development, asset-aware scheduling, and event-driven pipelines.

dbt Core and dbt Cloud specialist for model development, testing, macros, and project management.

PySpark and Spark SQL specialist for distributed data processing at scale.

Analyze DBSQL queries, including SQL embedded in notebooks (`spark.sql(...)`, `%sql` cells), for anti-patterns, lint issues, and performance problems, using Databricks-specific dialect and platform…

Creates dbt models following project conventions. Use when working with dbt models for:

Debugs and fixes dbt errors systematically. Use when working with dbt errors for:

Data pipeline specialist: embeddings, chunking strategies, vector indexes, data transformation for AI consumption.

This skill should be used when a story has been captured but content direction is unresolved - the spark describes WHAT to build structurally but not WHAT goes in it. Reads the story spark, splits…

This skill should be used when the user asks to "brainstorm", "explore options", "get creative", "what if we...", or selects "Let's get creative" during story or cycle creation. NOT for a bare vague…

Turn raw research, notes, links, or a conversation into a polished, shareable Sparkpage-style report page with an executive summary, structured sections, tables, and citations.

Turn research, a document, or a rough idea into a presentation-ready slide deck outline — narrative arc, slide-by-slide content, speaker notes, and visual direction — ready for AI Slides or manual…

Lightweight fixes and quick tweaks

Apply airflow dag patterns in reproducible local data workflows with version-aware APIs, explicit assumptions, and validation. Use when the user chooses airflow dag patterns or its strengths fit the…

Cursor rules for PySpark ETL development with code style, joins, window functions, map operations, and Iceberg patterns.

Cursor rules for Snowflake Cortex AI Functions (AI_COMPLETE, AI_CLASSIFY, AI_EXTRACT, etc.) and Cortex Search for RAG applications.

Cursor rules for Snowflake SQL, data pipelines (Dynamic Tables, Streams, Tasks, Snowpipe), semi-structured data, Snowflake Postgres, and cost optimization.

Generate SQL queries from natural language descriptions. Supports BigQuery, PostgreSQL, MySQL, and other dialects. Reads database schemas from uploaded diagrams or documentation. Use when writing…

Draw cloud architecture diagrams for AWS, Azure or GCP with the official provider icon sets, using TerraVision. Use whenever the user asks to draw, diagram, sketch or visualize a system on AWS,…

妙搭（Spark/Miaoda）应用开发与托管：应用创建、本地全栈开发、云端生成迭代、创意设计（UI mockup / 可交互原型 / 线框图 / 落地页 / 仪表盘 / 幻灯片 deck / 视觉探索）、AI相关能力和飞书平台能力或者其他外部能力集成、日志/Trace/监控指标/PV/UV…

Use whenever datasets, cloud storage buckets, or data pipelines are mentioned — creating, saving, querying, listing, exploring, deleting, or processing data in S3, GCS, Azure Blob, or local storage.…

Telemetry and analytics — event tracking, player behavior, A/B test frameworks, dashboard specification, data pipelines.

Data lake, analytics, and ETL workflows with S3 Tables, AWS Glue, and Athena.

Geospatial SQL skill for BigQuery and Snowflake. Discovers warehouse schemas, writes cost-safe SQL with mandatory dry-run, validates results, and optionally renders an interactive map.

Produces distinctive production UI and screenshot-grounded visual polish. Use when building or restyling pages, components, or typography. Not for backend logic or data pipelines.

Contributor workflow for the QuantMind codebase. Covers contributor setup (environment + hooks), filing issues, commit format, pull request format, and component development across quantmind/…

Upgrade an AWS Glue ETL job from Glue version 0.9 or 1.0 to Glue 4.0. Runs the job against Glue 4.0, diagnoses failures against a known breaking-change catalogue, patches the job script and…

Benchling R&D platform integration. Access registry (DNA, proteins), inventory, ELN entries, workflows via API, build Benchling Apps, query Data Warehouse, for lab data management automation.

Official NVIDIA-authored guidance for NVIDIA cuDF GPU DataFrames, pandas acceleration, dask-cuDF, ETL, joins, groupby, CSV/Parquet I/O, nullable semantics, and multi-GPU DataFrame workloads.

Use when the user asks for any draw.io diagram — cloud architecture (AWS, Azure, GCP, Databricks, multi-cloud, hybrid/DR, landing zone, VPC/network, serverless, data pipeline) or a business process…

Build interactive, click-through architecture diagrams for software systems — a self-contained single HTML file with animated step-by-step flows, mode toggles (dev/prod, offline/online, v1/v2),…

Build ETL pipelines, data warehouses, and streaming architectures. Implements Spark jobs, Airflow DAGs, and Kafka streams. Use PROACTIVELY for data pipeline design or analytics infrastructure.

Data analysis expert for SQL queries, BigQuery operations, and data insights. Use proactively for data analysis tasks and queries.

Manage Jitsu data pipelines: destinations, streams, connections, functions, live events.

Expert knowledge for Azure Anyscale On Azure development including limits & quotas, security, configuration, integrations & coding patterns, and deployment. Use when registering AKS as Anyscale…

Expert FinOps guidance for cloud, AI, SaaS, and data-platform spend. 21 reference files spanning AWS (CUR + Data Exports for FOCUS 1.2, Savings Plans incl. SageMaker AI + Database), Azure…

Monte Carlo Agent Toolkit gives AI coding agents data observability superpowers across the full data lifecycle. Before querying a table, the agent checks freshness, alerts, and monitor coverage.…

Data engineering patterns for ETL pipelines, data warehousing, Apache Spark, and data quality validation