Vinoth Chandar @byte_array

It’s snowing in the San Francisco summer.. Most companies offload their Snowflake ETLs to spark; We launched Quanton on Snowflake. So, you can finally run real cutting-edge Apache Spark, inside Snowflake.. For the what, how, why: checkout Kyle Weller ‘s blog here 👉 onehouse.ai/blog/real-apac…

Onehouse isn’t at Snowflake Summit this year (let’s not go into reasons). But someone should be having the real Spark-on-Snowflake conversation — so we showed up anyway. On a truck 🚛 Here’s why: - Real Apache Spark — inside Snowflake containers - Iceberg writes up to 4× faster - No lift-and-shift, no second platform — especially if your data already lives in parquet/iceberg/delta/hudi - Sitting on unspent Snowflake credits? Use them for real Spark instead of paying yet another vendor - And since Snowpark containers can be cheaper than Snowflake warehouses, you could spend up to 65% fewer credits We didn’t build Quanton to win a benchmark slide. We built it so data teams don’t have to move their data, double their bill, or pick a side in someone else’s platform war just to run a workload. At Summit? Wave at the truck. Not here? quanton.dev/docs/guides/sn….

More honesty is good. 👍🏻 Its very frustrating when AI (also) lies about work.

Claude @claudeai

7 days ago

Introducing Claude Opus 4.8: it builds on Opus 4.7 with sharper judgment, more honesty about its own progress, and the ability to work independently for longer than its predecessors. Available today at the same price.

4K 9K 67K 15.0M 8K

Picture your lakehouse bogged down by updates at 10TB scale 📊? Core issue: Finding records without scanning full table. Hudi's file groups route same-key records to one group with base/log files 🔑. Updates are bound to that group 🚀—skip full searches. Wins: • Bounded write locality • Limited scans for merge reads/compaction At scale, this stabilizes ingestion vs. messy maintenance or sluggish MERGE INTO 🛠️.

Building indexes on a 100 TB Hudi table without blocking writes. The naive approach: stop writes, build the index, resume. Hours of downtime. Hudi async indexing: bootstrap index, then converge , high concurrency ↓

Check out here: quanton.dev ;

Quanton matches Databricks Photon on reads and is much faster on writes—up to 4x faster/cheaper Spark at this price. DM or leave comments - for feedback!

📉 In 2025, Onehouse's revenue growth was affected—not from churn or slower market, but because we rolled out Record-Level Indexing (RLI) across our biggest pipelines. RLI (used at Uber-scale with Apache Hudi) is key reason Onehouse is the most efficient option for large-scale ingestion/ETL, often cutting Spark ingest/ETL compute 50%+. As customers adopted it, metered compute hours dropped ~50% even as data grew. This reveals a deep problem within the industry and also why platforms don't optimize harder with techniques like indexing. incentives! 💰 For EMR/Databricks Classic: faster jobs → lowers revenue. Not good for vendor. 🏦 For Databricks serverless/Snowflake: faster jobs → better vendor margins. No direct benefits to user. 🔄 What did we do?: volume-based pricing for Spark + SQL pipelines on Quanton.dev running in the user's cloud. ⚡ We make Quanton faster → directly lowers user's compute spend → Onehouse earns based on different metric: GB processed/month. #ApacheHudi #ApacheIceberg #DeltaLake #Lakehouse #CloudOptimization #DataEngineering

Google just stepped on the gas and left the keyword search era in the rearview mirror.

Logan Kilpatrick @OfficialLoganK

2 weeks ago

Today we are starting to roll out the biggest upgrade to the Google Search box in over 25 years — now completely reimagined with AI, along with Gemini 3.5 Flash as the new default model for AI mode users globally!

253 283 4K 1.4M 1K

You can take our engine for a spin here: quanton.dev

⚡ Speed is Iceberg’s biggest Achilles’ heel. Most teams chase Velox or Gluten for Spark speed. Yet the real wins sit in fast I/O and layers untouched. We’ve spent years making lakehouse pipelines fast—but this isn’t a table-format debate. It’s slow I/O, shuffles, and reprocessing driving costs up. Speed is a stack: ⚡ SIMD 📊 Columnar processing 🔍 Query plans 💾 Fast I/O 🗄️ Metadata 📐 Storage layout 🔗 Indexes 🔄 Merging Full-stack engine: 3–4× faster than OSS Spark on Iceberg, better price/perf than Photon, no lock-in.

Good to hear from you (long time :)) I think data scanned is actually directly relate-able. I would not say the same thing around the MAR metric. "data processed" or "records scanned" don't mean much to business user, agree. But, the platform efficiency is something the technical buyer/user of the tech needs to solve. Not claiming to solve that. My claim is : data volume is more directly relatable to the technical users, than deriving the compute hours - then having to balance platform markups against compute hours. The flaw in BigQuery, Athena and Fivetran model is that compute still runs on vendor side. So, users don't find it that attractive. We have actually landed volume-based pricing in quanton.dev with compute now running in customer's own accounts. This now decouples what you play for a fast engine and the underlying compute cost you incur. We can keep making Quanton faster without cutting our own revenue (incentives are aligned)

🧠 Anthropic and OpenAI may have taught Databricks and big data vendors a pricing lesson. 🤖 Frontier models price inference by tokens, not GPU FLOPs. it’s intuitive: bigger context or more work costs more. Cloud data platforms still price by compute: DBUs, EC2 hours, slot time. Ask: “How many DBUs will this query use on Databricks vs EMR?” 🤷 Ask: “How much data will it scan?” most engineers can answer fast.

Compaction isn't just housekeeping 🧹—it's deferred write work in Merge-on-Read systems 📝. Append changes to logs for fast ingestion 🚀, then compaction folds deltas into optimized base files 🔄. Real question: When to pay back that deferred cost? ⏳ It affects: - Ingest latency ⏱️ - Read performance 📈 - Storage efficiency 💾 - Background resource usage ⚙️ Hudi makes this tradeoff explicit, treating compaction as a first-class table service.

What does an ideal pricing model look like that balances incentives for everyone and is actually simple to reason about?

Data engineers: hours tuning Spark jobs, yet Hudi, Iceberg, or Parquet reads still take minutes? We've all been there. Most pipelines bottleneck pre-join. Details below 👇

Hudi's merge spill storage draws from BitCask 💡 BitCaskDiskMap appends values to a single log file. In-memory ConcurrentHashMap tracks key metadata: offset, size, timestamp—keeping memory low for quick offset reads 📊 ThreadLocal BufferedRandomAccessFile handles enable lock-free concurrent reads 🔄 CRC checksums on writes, validated on reads, catch disk corruption early ✅ ThreadLocal compression caching reduces allocations in heavy spills ⚙️ Go-to backend for ExternalSpillableMap: handles MOR merge spills with fast appends and random access 🚀

Excited to moderate the "Open by Design" panel at #OpenXData2026! 🚀 We’re bringing together leaders from across the @apachehudi , @ApacheIceberg , @prestodb , and @ApacheFlink ecosystems to talk about why community is the true foundation of open data.

Register free here: openxdata.ai

Everyone thinks data platforms are ready for AI agents. But they're built for human workflows, forcing a total rethink now. OpenXData, free virtual event, next Wed captures this shift. Last decade: platforms for dashboards, reports, pipelines. Now: AI agents operate continuously at machine speed. Talks show it: 📄 Formats like Lance push beyond structured 🗄️ Hudi for unstructured + vectors 💬 Context eng = data eng for agents 🤖 AI in Spark reshapes ops 🛠️ Tools clean PDFs Converging to: 👉 Context infra for AI My keynote connects dots on lakehouse evolution.