◈ Homepage — https://www.startree.ai/Skip to main content Skip to footer May 20 - Webinar - Iceberg Performance Benchmark: StarTree vs. Trino vs. ClickHouse : RSVP Here Product Solutions Token-Level Observability for LLMs: How Together AI Does It BurdaForward: Analyzing Quality News Performance Using Real-Time Analytics Webex Meets Unprecedented Analytics Demand with Apache Pinot Dialpad Powers Real-Time Customer Intelligence with StarTree Cloud Taming the Kafka Firehose: How CrowdStrike is Scaling Real-Time Security Analytics with Pinot Token-Level Observability for LLMs: How Together AI Does It BurdaForward: Analyzing Quality News Performance Using Real-Time Analytics Webex Meets Unprecedented Analytics Demand with Apache Pinot Dialpad Powers Real-Time Customer Intelligence with StarTree Cloud Docs Resources Pricing Login Talk to a Pinot expert The Platform for Continuous Insights StarTree, powered by Apache Pinot, turns raw data streams—like clickstreams, logs, vector embeddings, and sensor data—into real-time analytics. It powers customer-facing data products, agent-facing data products, and operational intelligence. WATCH THE RECORDING Unlocking Revenue-Critical workloads on Iceberg Watch our recent webinar on how to leverage a purpose-built execution layer to achieve deterministic SLAs directly on Iceberg tables. Eliminate shadow stacks and restore architectural coherence for revenue-critical insights. Watch Now Proven with the most demanding use cases How Grab Built a Next-Gen Observability Stack for Its Rideshare Marketplace One of Southeast Asia’s largest makers of ‘super apps’ – is powering real-time observability in its rideshare marketplace with Apache Pinot. Its integrated platform supports operations across more than 300 cities with one-second response times. In this talk from Real-time … Learn more 10 Million Requests per month 1 second Query Latency (p95) 90% Growth Over six months Webex Meets Unprecedented Analytics Demand with Apache Pinot Seeking a real-time analytics solution to meet spiking customer demand for insights in the pandemic era, Webex chose Apache Pinot over Clickhouse and Elasticsearch. Learn more 100 tb data daily 100+ production nodes 4x faster queries Token-Level Observability for LLMs: How Together AI Does It As large language models (LLMs) become core infrastructure, a new engineering challenge is taking shape: capturing how these models are used—and how they behave—as it happens. It’s no longer sufficient to count API calls or log failures after the fact. … Learn more Sub-second Query Latency High-cardinality slicing and dicing 10 second freshness windows How Grab Built a Next-Gen Observability Stack for Its Rideshare Marketplace One of Southeast Asia’s largest makers of ‘super apps’ – is powering real-time observability in its rideshare marketplace with Apache Pinot. Its integrated platform supports operations across more than 300 cities with one-second response times. In this talk from Real-time … Learn more 10 Million Requests per month 1 second Query Latency (p95) 90% Growth Over six months Webex Meets Unprecedented Analytics Demand with Apache Pinot Seeking a real-time analytics solution to meet spiking customer demand for insights in the pandemic era, Webex chose Apache Pinot over Clickhouse and Elasticsearch. Learn more 100 tb data daily 100+ production nodes 4x faster queries Token-Level Observability for LLMs: How Together AI Does It As large language models (LLMs) become core infrastructure, a new engineering challenge is taking shape: capturing how these models are used—and how they behave—as it happens. It’s no longer sufficient to count API calls or log failures after the fact. … Learn more Sub-second Query Latency High-cardinality slicing and dicing 10 second freshness windows How Grab Built a Next-Gen Observability Stack for Its Rideshare Marketplace One of Southeast Asia’s largest makers of ‘super apps’ – is powering real-time observability in its rideshare marketplace with Apache Pinot. Its integrated platform supports operations across more than 300 cities with one-second response times. In this talk from Real-time … Learn more 10 Million Requests per month 1 second Query Latency (p95) 90% Growth Over six months See more case studies APACHE PINOT For low-latency, high-concurrency analytics FAST Sub-second analytic queries Blazing fast query speeds through innovative indexing techniques that send near instant responses — even at high volume &emdash; with a wide variety of data types including JSON, vector, timestamps and geospatial. More FRESH Insights on live data Continuous ingestion keeps data flowing. Indexes are updated as data lands – in real-time. Pinot is proven to ingest 1M events/s, while also maintaining high throughput for queries. More HIGH CONCURRENCY Extreme queries per second Support rich user-facing analytic applications. Pinot is capable of serving hundreds of thousands of simultaneous queries across huge and changing datasets without performance bottlenecks. More AFFORDABLE Low compute and storage costs Pinot has been engineered to scan and move less data, so it’s faster AND cheaper. Innovative optimizations such as data partitioning strategies, segment assignment strategies, and smart query routing reduce compute and storage costs. More Apache Pinot is a rapidly growing and thriving open-source project. More STARTREE CLOUD Fully managed, enterprise-grade StarTree extends Apache Pinot into a managed data platform with enterprise-grade features so you can focus on insights, not infrastructure. StarTree adds security, scalability, and many additional capabilities: Interactive queries on the data lake Precision fetch allows for interactive query on data in object storage — directly, and without large data transfers. More Easier Administration Eliminate unnecessary infrastructure costs during idle times with autoscaling; also, ease of cluster, instance, and storage management.  More Scalable Upserts Off-heap upsert functionality can handle updates to billions of primary keys per server without sacrificing performance. More Enterprise Security Additional security features, including RBAC, advanced data encryption, single sign-on, and SOC 2 and ISO 27001 compliance ensure your data is safe and protected. More Improved data management StarTree's Data Portal UI adds capabilities and makes it easier to ingest and manage data. Evolve schema, tune performance and backfill data without disrupting operations. More Improved data management Connect with other enterprise data sources, such as Confluent, Snowflake and Tableau with additional StarTree integrations. More DEPLOYMENT OPTIONS DBaaS, BYOC or BYOK? StarTree Cloud offers public SaaS (Software as a Service) and private BYOC/BYOK deployment options. For more information about our public or private SaaS offering, schedule a demo with our StarTree solutions architect team. STARTREE CLOUD SaaS Get started quickly with the fully managed, StarTree Cloud service. Ideal for organizations that want zero ops overhead and rapid time-to-value. Setup is simple, leaving you to focus on your data, your schema, and queries. BRING YOUR OWN CLOUD BYOC For customers who prefer dedicated infrastructure and deeper integration with enterprise systems, a Bring Your Own Cloud (BYOC) model includes VPC peering, custom SLAs, and enhanced observability. BRING YOUR OWN KUBERNETES BYOK For highly regulated environments, StarTree can also be deployed behind your firewall—offering full control while still leveraging the power and capabilities of StarTree Cloud. Deployment Options: Learn More The benefits of migrating to StarTree and Pinot center around three areas: query performance, SLAs, and cost. With Pinot, we are now in the subseconds to milliseconds range, the higher query concurrency means we can serve more customers faster, and we’ve been able to reduce our infrastructure costs. Joanes Espanol CTO and Co-Founder GET THE GUIDE Real-time analytics at scale Leading organizations like Uber and Stripe are harnessing real-time insights to efficiently power customer-facing data products. This comprehensive report details how they do it. Download your Digital Copy Today First Name* Last Name* Company Name* Business Email* I have questions for a product expert. I agree to the StarTree Community Terms and Conditions.* PLATFORM Apache Pinot StarTree Cloud StarTree ThirdEye Deployment Options Pricing SOLUTIONS Real-time Analytics Case Studies Observability Fraud Detection Customer-Facing Analytics FOR DEVELOPERS Docs Community Support Releases RTA Summit ABOUT About Careers Partners Press & Media Contact Legal Sign up for our newsletter... © 2025 StarTree Inc. All rights reserved. Trademark Policy Privacy Policy Terms of Use Responsible Disclosure ◈ Interior Pages — 3 pages crawledStartree release notes - StarTree Docs Skip to main content StarTree Docs home page Search... ⌘ K Ask AI Go to StarTree.ai Talk to us Talk to us Search... Navigation Home StarTree Cloud ThirdEye Apache Pinot Recipes AI API Reference Community Blogs Videos Get Started StarTree Cloud What is Real-Time Analytics? Ingestion Overview Connectors Data Modeling Additional Configuration Advanced Operations Recipes StarTree Iceberg/S3 Tables Onboarding via Data Portal Observability Data Type Mapping Indexes Data and Index caching Query Data Query Languages Query Interfaces Query Functions Advanced Operations Query Analyzer StarTree MCP server Manage Data Indexes Upserts Off-Heap Dedup Cloud Tiered Storage Hybrid Tables Data Operations Recipes Visualize Data Tableau Superset Manage Security Overview Authorization API Token Management Pinot Audit Logging Configure IdP Release Notes Data Portal 0.15.0 0.14.0 Older Releases Reference Glossary Cluster Health StarTree Cloud V2 Benefits On this page StarTree release notes Startree release notes Documentation Index Fetch the complete documentation index at: https://docs.startree.ai/llms.txt Use this file to discover all available pages before exploring further. ​ StarTree release notes 0.12.0 : February 2026 0.11.1 : September 2025 0.11.0 : July 2025 0.10.1 : February 2025 0.10.0 : November 2024 0.9.0 : June 2024 0.8.1 : April/May 2024 0.8.0 : February 2024 0.7.2 : December 2023 0.7.1 : November 2023 0.7.0 : August 2023 0.6.1 : April 2023 0.6.0 : February 2023 0.5.0 : June - August 2022 0.4.0 : April - May 2022 Was this page helpful? Yes No ⌘ I StarTree Docs home page x youtube github linkedin Trademarks Privacy Terms of Use x youtube github linkedin x youtube github linkedin Powered by This documentation is built and hosted on Mintlify, a developer documentation platform Assistant Responses are generated using AI and may contain mistakes. Download Recipes - StarTree Docs Skip to main content StarTree Docs home page Search... ⌘ K Ask AI Go to StarTree.ai Talk to us Talk to us Search... Navigation Recipes Download Recipes Home StarTree Cloud ThirdEye Apache Pinot Recipes AI API Reference Community Blogs Videos Recipes Download Recipes Batch Ingestion Streaming Ingestion Transformation Functions Deep Storage Upserts Real-Time-Offline Job JSON Documents Geospatial Merge and Roll Up Clickstream Analytics Dashboard Other On this page Clone the pinot-recipes repository using SSH Clone the pinot-recipes repository using HTTPS Recipes Download Recipes Documentation Index Fetch the complete documentation index at: https://docs.startree.ai/llms.txt Use this file to discover all available pages before exploring further. Recipes help you learn how to solve common problems with Apache Pinot. To download Apache Pinot recipes, do one of the following: Clone the repository using SSH Clone the repository using HTTPS If you don’t have a Git client or don’t want to clone the repository, click to download a zip file that contains the recipes. ​ Clone the pinot-recipes repository using SSH To clone the pinot-recipes repository using SSH, do the following: Set up SSH keys associated with your GitHub account on your local machine. For more information, see Connecting to GitHub with SSH . Run the following command: git clone [email protected]:startreedata/pinot-recipes.git ​ Clone the pinot-recipes repository using HTTPS To clone the pinot-recipes repository using HTTPS, do the following: Run the following command: git clone https://github.com/startreedata/pinot-recipes.git At the prompt, enter your GitHub username and a personal access token. If you forgot your GitHub username, see Remembering your GitHub username or email . If you need a personal access token, see Creating a personal access token . Was this page helpful? Yes No CSV with Column Spaces ⌘ I StarTree Docs home page x youtube github linkedin Trademarks Privacy Terms of Use x youtube github linkedin x youtube github linkedin Powered by This documentation is built and hosted on Mintlify, a developer documentation platform Assistant Responses are generated using AI and may contain mistakes. Taming the Kafka Firehose: How CrowdStrike is Scaling Real-Time Security Analytics with Pinot | StarTree Settings --> Settings --> Skip to main content Skip to footer May 20 - Webinar - Iceberg Performance Benchmark: StarTree vs. Trino vs. ClickHouse :  RSVP Here Product What is Apache Pinot? An open-source database for real-time analytics at scale and speed. More Apache Pinot on StarTree Cloud StarTree extends Apache Pinot as a fully managed service, with enterprise features and improved performance. Overview Architecture Features Security & Compliance Releases Deployment Options Iceberg Tables StarTree Cloud now offers reliable, low-latency query on the data lake. More Solutions Learn about the foundations behind customer-facing dashboards, agent-facing analytics, and observability systems. More By Industry Finance Healthcare Gaming Retail Transportation & Delivery By Use Case Customer Facing Analytics Observability Machine Learning Fraud Detection Interactive Analytics for the Lakehouse Case Studies (See all) Webex Meets Unprecedented Analytics Demand with Apache Pinot Dialpad Powers Real-Time Customer Intelligence with StarTree Cloud Taming the Kafka Firehose: How CrowdStrike is Scaling Real-Time Security Analytics with Pinot Token-Level Observability for LLMs: How Together AI Does It BurdaForward: Analyzing Quality News Performance Using Real-Time Analytics Docs Request a Trial Start with StarTree Cloud! Our onboarding team will help you succeed! Get Started Documentation Ingestion Query Data Indexes Tiered Storage Security Releases API Reference Get Involved Community Support Apache Pinot 101 Pinot Recipes Resources Grafana has become the de facto standard for visualizing operational metrics, thanks to its rich ecosystem and intuitive PromQL-based query… Resources Blogs In the News Events & Webinars Videos Case Studies Comparisons Pinot vs Druid vs Clickhouse Pinot vs ElasticSearch Pinot vs pre-aggregated KV Stores Pinot vs Time-Series Databases Pinot vs StarTree? Pricing Login Book a Demo Login Talk to&nbspa Pinot expert CASE STUDY Taming the Kafka Firehose: How CrowdStrike is Scaling Real-Time Security Analytics with Pinot CrowdStrike leverages Apache Pinot’s real-time distributed OLAP datastore to support security analytics handling 25k QPS across 10 clusters. 120,000 events per second 25,000 queries per second 5 billion events (on one table) In this talk at RTA Summit 2025, Yusuf Kulah shares how they do it! CrowdStrike’s backend runs on dozens of microservices, many of which consume a shared firehose of events via the Kafka distributed event streaming platform, said Yusuf Külah, a CrowdStrike senior engineer, in a technical session at the Real-Time Analytics Summit hosted by StarTree. Kafka acts as a central messaging system, broadcasting a stream of data like system metrics from endpoint agents. Managing variable loads The problem facing Crowdstrike was when Kafka topics experienced highly variable loads, often spiking when new customers came on board or when malware caused a surge in system events. Microservices like front-end request handlers must respond quickly in such situations, while others take longer to digest data. To keep service-level objectives (SLOs) intact, the team built a Pinot-powered monitoring layer that tracks event volume in real-time and publishes signals to a data bus when thresholds are crossed. This “traffic cop” approach helps services implement throttling or other mitigation plans dynamically. Supporting security analytics Using the same Kafka event stream, CrowdStrike applied Pinot to a second challenge: detecting data exfiltration. Crowdstrike aggregates and deduplicates events like file transfers or outbound network activity in real-time to ensure analysts get a clean feed of what’s leaving the system and where it’s going. Security analysts can monitor a dashboard to slice and dice data by destination, source, or other dimensions. This enables them to quickly pinpoint, for example, a rogue internal user shipping data to an unapproved endpoint. Wrestling with Protobuf complexity CrowdStrike’s events are encoded in Protobuf, a compact data serialization format that lets them define structured data using schemas, including nested types and enumerations. While Protobuf is great for performance and consistency, it’s less effective when schemas are evolving, deeply nested, and not uniform across environments. A bigger problem was that Protobuf changes forced restarts of Pinot servers, since Pinot had to reload descriptors with each change. Parsing complex Protobufs was slow, especially when only part of the message was relevant. Crowdstrike’s solution was to introduce a preprocessing layer. This component takes complex Protobuf events, extracts what’s needed, and emits a “lean” JSON version aligned with Pinot’s schema. It offloads heavy parsing, improves performance, skips unneeded fields, and avoids Pinot restarts. The tradeoff is that Kafka carries a duplicate lean event stream, which adds storage cost. Crowdstrike determined that benefits outweighed the bloat, however. Kafka’s dynamic addressing challenge In a dynamic infrastructure, Kafka message broker addresses change frequently, challenging Pinot to keep up. Failure to refresh broker lists in real-time risks ingestion delays or data loss, which are unacceptable in security analytics. DNS-based discovery was an obvious solution but not an option due to internal security policies, so CrowdStrike built an automated broker updater. This keeps Pinot’s configuration in sync with Kafka’s current state, ensuring smooth ingestion even as the cluster evolves. As a bonus, the updater mechanism also helped prevent configuration drift. If a Pinot table’s settings deviated from the official configuration, the updater would automatically realign it. Turning point-in-time queries into feeds Pinot is fast, but single queries are just snapshots. CrowdStrike needed a continuous data feed for some services, especially those that make decisions based on trends, such as a spike in data egress volumes over a defined period of time. Running hundreds or thousands of point-in-time queries from microservice replicas would have been wasteful and redundant, so the CrowdStrike team built a query scheduler to convert snapshots into continuous feeds. The scheduler deduplicates queries, reduces processing load, and enables real-time context. The big picture The system comprises four layers as illustrated: Data sources are primarily Kafka, but also include some batch feeds. The Preprocessor parses Protobufs and emits lean JSON events. The Apache Pinot cluster is the core analytics engine. It ingests lean events into real-time tables and optionally into batch-aggregated offline tables for long-term queries. Data consumers are the downstream services and dashboards. Some query Pinot directly, while others rely on the continuous feed via the query scheduler. Pinot is deployed in a multi-tenant setup, with some servers dedicated per tenant depending on the workload. This design balances performance and isolation. The results CrowdStrike’s Pinot deployment has been able to handle whatever real-time data is thrown at it. A single table replica handles up to 120,000 events per second. One table has seen over 5 billion ingested events. The busiest table supports up to 25,000 queries per second. All of this runs across five global production environments and more than 10 Pinot clusters. CrowdStrike’s real-time analytics system isn’t just fast; it’s resilient, scalable, and purpose-built for high-stakes cybersecurity work. From preprocessing complex data to automating infrastructure adaptation, the team’s approach demonstrates how thoughtful engineering can turn raw event firehoses into actionable insights. Learn more StarTree provides Apache Pinot as a managed service with many extended capabilities. Book a meeting with one of our experts to learn more a