ABC @Ubunta

Talked to dozens of young EU devs lately. The brutal job market has them desperate — stacking AI subscriptions (OpenAI, Claude, Cursor) and starring repos they never open. It feels like progress. It isn't. Buying tools isn't building skill, and chasing the trend isn't knowing where it's going. The real question: what does the market want now, and where are dev jobs even heading?

which llm model I reach for Data Engineering, and when: Claude Opus 4.8 — building from zero - deterministic data apps - pyspark / distributed pipelines - hand-optimized DB access libs for big reads OpenAI Codex 5.5 — surgery on what exists - bolting security onto live pipelines - single-node duckdb work - refactoring tangled multi-connector pipelines

LLM releases now feel like Windows updates. Claude, OpenAI, DeepSeek — everyone is dropping models so frequently that you barely finish testing one before the next one arrives. And just like Windows updates, the changelog sounds exciting… …but after restarting, everything feels mostly the same.

I built a Clinical Trials AI agent on top of the entire ClinicalTrials.gov database (AACT — 50+ tables, 12M+ rows), and I want to share a few things I learned, because most of them surprised me. The goal: not another chatbot that "talks to a database," but an agent that could actually reason over a massive, real-world dataset and be useful. And I vibe-coded the whole thing — directing AI to architect and build it, while I held the design decisions, the data model, and the deployment. 1. The hard part isn't the agent. It's the data. You can wire up a model in an afternoon. But 12M rows don't answer questions quickly or cheaply by accident. Almost all the real effort went into the boring layer underneath: a two-DB split (a read-only pool just for AACT), materialized views + hand-tuned SQL instead of an ORM fighting the schema, versioned snapshots with automated sync as new dumps land, and caching so hot queries stay cheap. The agent is the easy 10%. The data engineering is the 90% nobody films. 2. You don't need an agent framework. You need a clean tool contract. No LangChain, no orchestration library. Just a tiny registry: defineTool() to declare a tool, runTool() to call one. The thing I'd underline for anyone building agents — every call goes through the same pipeline: schema-validated (Zod) → policy-checked → executed → audited. That one invariant is worth more than any framework. The control loop stays mine, and adding a capability is one file. 3. Give the model a toolbelt, not a database. Instead of raw SQL, I gave it nine purpose-built tools — trials_search, study_get, eligibility_lookup, feasibility,  competitive_landscape, safety_profile, and more. Each encodes how a human actually thinks about clinical trials, and the model composes them. You're not building a query interface, you're building the agent's vocabulary for the domain. 4. Decouple from the model early. Everything goes through one getModel(role) factory — Anthropic + OpenAI today, switchable per task. Adding Bedrock, Azure, or a local model is one file, zero refactor. Models change every few months now; your architecture shouldn't care which one is winning this week. 4. Decouple from the model early. Everything goes through one getModel(role) factory — Anthropic + OpenAI today, switchable per task. Adding Bedrock, Azure, or a local model is one file, zero refactor. Models change every few months now; your architecture shouldn't care which one is winning this week.  5. Vibe-coding is a real skill, and it's not "typing less." The work wasn't writing code. It was steering an AI to produce a clean, production-shaped system — knowing what good architecture looks like, catching when it drifts, then getting it deployed: containerized, health checks, telemetry on every call.

The hardest problem in AI-assisted Data Engineering isn't token burn. It's measuring token burn against actual engineering outcomes. You can burn millions of tokens and end up with a solid pipeline — or complete garbage. The real trap: letting AI talk directly to your database. That's a token-burn accelerant. AI shouldn't freely explore your DB. Give it bounded context, clear contracts, safe APIs, schema summaries, query limits, and measurable outcomes. Otherwise you're not doing AI-powered Data Engineering. You're just paying for confusion at scale.

Tokenmaxxing won't save bad Data Engineering. Weak foundations will sink it faster. If you ask AI to build entire pipelines without: - clear desired outcomes - a proper dev environment - defined tech choices and responsibilities - a solid design structure …you'll burn infinite tokens and still ship garbage pipelines. AI doesn't replace engineering fundamentals — it amplifies them. Strong software practices, local testing, and safe environments matter more now, not less. If your data pipelines hit prod directly, don't blame AI. AI just exposed what was already broken.

Hard truth for Data Engineers: Knowing your tools, writing ETL, and shipping PySpark jobs is no longer a moat. It's the baseline — and the baseline is being automated. The next wave isn't "Data Engineer 2.0." It's full-stack engineers who use AI to solve data challenges faster than any team could last year. The ask isn't more data engineers. It's more AI data automation.

5 places AI automation still doesn’t fully belong in large-scale Data Engineering platforms: 1. Domain logic & sanity checks — too business-specific 2. Infra & secrets — no-go zone 3. Pipeline tests — no intent = fake confidence 4. DB migrations — schema changes, backfills, and rollbacks need humans 5. Prod decisions — reruns, data loss, and access changes need accountable owners AI assists. Humans own intent, controls, and risk.

A year ago I wouldn't trust AI with a JOIN. Last week it built a data pipeline in SQL and Python that's running in production, no issues. Data engineers should stop asking: – Can AI write production-grade code? – Will it replace me? – Should I bother learning AI-native tools? The shift already happened. Now it's just about who keeps up.

I’m honestly unsure which part of Data Engineering cannot be automated with GenAI anymore. That does not mean you don’t need data engineers. But it does mean you probably don’t need the same size team as before. In many cases, maybe not even half the team you needed earlier.

Healthcare AI is forcing a rethink of how RAG systems should actually work. Traditional vector RAG is great for FAQs, support systems, and broad semantic lookup. But once you move into clinical protocols, SAPs, regulatory submissions, research papers, or evidence packages, the retrieval problem changes completely. The challenge is no longer: "find semantically similar text." It becomes: – navigating document hierarchy – reasoning across sections – preserving traceability to source pages – and avoiding retrieval that is "similar" but contextually wrong A clinician reviewing a protocol does not think in chunks and embeddings. They navigate endpoints, inclusion criteria, appendices, statistical methodology, references, and cross-document relationships. Retrieval systems should mirror that workflow instead of flattening everything into vector similarity. This is why I've been experimenting with approaches like PageIndex (github.com/VectifyAI/Page…). What I find interesting is not the "vectorless" angle itself. It's the shift toward reasoning-based retrieval using hierarchical document structures and tree navigation that behaves much closer to how domain experts actually read long documents. I don't think vector RAG disappears. It still solves many problems well. But for long-form, structured, regulated domains like healthcare, I increasingly think the future is hybrid: vector retrieval + reasoning-based document navigation working together in the same platform. Then let healthcare professionals judge which outputs are actually more trustworthy, traceable, and clinically useful. Because in regulated AI systems, retrieval quality is not just a UX feature. It's part of the safety layer.

Top dangerous things to do in Data Engineering 1. Backups inside the same blast radius. Same region, same admin key, same failure path. That is not disaster recovery. 2. Letting AI touch production directly. AI can draft deployment code. It should not control your production cluster. 3. Connecting MCP servers without governance. Every new tool connection is a new permission boundary, audit gap, and attack surface. 4. Running without serious observability Silent failures, data drift, runaway costs, and wrong outputs are worse when nobody is watching. Most data engineering disasters start with one thing: too much access and too little control.

Through recent conferences and conversations, two approaches to GenAI keep showing up. On one side, enterprises are still debating the risks and relevance without ever touching it. On the other, teams are buying every tool in sight, burning budget at speed, then concluding: “AI doesn’t work.” Different teams. Same mistake. One is fear without data. The other is spending without strategy. Both skip the only step that actually matters → small, deliberate experiments. Your environment. Your data. Your constraints.You don’t get to an opinion on GenAI by only reading about it. You don’t get to ROI by buying your way there. You get there by building something small, watching where it fails, and paying attention to why.

The way we build Data Pipelines in regulated healthcare is changing. AI is no longer a downstream consumer — it is becoming a component inside the pipeline itself. And that is where the architecture gets interesting. The old shape was familiar. Sources → ingest → transform → warehouse → BI. Never fully deterministic — late-arriving data, schema drift, manual labeling all leaked in — but the failure modes were known and the fixes were boring. Healthcare data was messy but the pipeline behavior was predictable. AI changes the shape. An LLM doing chart abstraction mid-DAG. An agent selecting a cohort definition. A RAG call enriching a record before it lands in the warehouse. Now the pipeline has a new class of failure — silent semantic corruption, non-reproducible outputs, cost blowups from agent loops. In a regulated environment, that is the whole problem. The discipline is simple. Not easy. Keep the pipeline deterministic. Let AI live only inside bounded, validated nodes. I keep going back to how the biodata community solved reproducibility — nf-core / Nextflow → DAG-first execution, content-addressed caching, resume-on-failure, containerized steps, provenance baked in. That mindset translates directly. I am building it now: - DAG as the backbone. Idempotent steps, content-hashed outputs. - A common data model as the semantic layer. Schema validation non-negotiable. - Provenance tracked per record, not per batch. - LLM nowhere near the orchestrator. Only inside scoped nodes — chart abstraction, endpoint adjudication drafting, protocol-to-SQL translation. - Every LLM output hits a deterministic validator before persistence. → Eval layer built before the agents. Clinician-labeled ground truth, re-run on every model bump. Then AI earns its place in the pipeline — and a regulator can still follow the trail.

Switching from Claude or Codex to a local coding model for data engineering makes a few things very obvious. The planning quality drops — less context carried across steps, weaker breakdown of problems, and more gaps in logic (especially around joins, transformations, and edge cases). Iteration also slows down a lot. What used to be quick back-and-forth becomes noticeably delayed, which affects how fast you can validate ideas. On top of that, the mac becomes the bottleneck. High resource usage leads to heating and throttling, and overall system responsiveness takes a hit. While local models reduce external dependencies, the current trade-off is lower reasoning quality and slower workflows, especially for non-trivial data engineering tasks.

If you’re still not convinced about using AI in your Data platform, think in simple risk strategy terms. Best case? - Big productivity gains Most likely? - Incremental but real improvements Worst case? - Some errors, extra validation Now be honest. If the most likely outcome already moves you forward,and the worst case is manageable, there isn’t much argument left. This isn’t AI hype — it’s basic risk/reward thinking. AI won’t replace data engineers. It shifts the work: from writing code → to validating and owning outcomes. The upside is asymmetric. That’s usually enough to act.

AI in a healthcare data platform is not a tooling problem. It's a governance problem. The moment GenAI enters a regulated platform, the platform changes shape. It stops being a system people query. It becomes a system that acts on its own interpretation of intent. That shift is where the discomfort starts. Traditional governance is hard, but deterministic. Access is defined, policies are enforced, lineage is tracked, changes are auditable. None of that is naturally guaranteed with AI. A model can generate queries you didn't anticipate, join datasets you never intended to combine, and be confidently wrong where correctness is non-negotiable. In healthcare, confidently wrong is not a bug. It's a compliance event. So I've stopped treating AI as a capability. I treat it as an untrusted layer. It should not directly access data. It should not directly touch infrastructure. It should stay at the level of intent, generating what should be done rather than doing it. Whatever it produces flows through the governed paths the platform already trusts — data contracts, policy checks, controlled execution, audit trails. This flips the usual architecture. Instead of wrapping governance around AI, governance becomes the system AI is forced to operate through. GenAI is probabilistic. Healthcare platforms are built on determinism and accountability. The real work isn't integrating AI into the platform. It's redefining the boundaries so AI can exist without weakening the guarantees the platform was built for. The answer is not more access for AI. It's stricter control over where AI is allowed to exist.

There are early signs that LLM coding performance might be slowing down. - The big claims around Claude’s new Mythos don’t really hold up x.com/elliotarledge/… - Claude Code quality drops noticeably during core working hours (there’s even a ticket ) github.com/anthropics/cla… - Cursor built a strong model on Kimi-k2.5 that’s already close to top-tier models - Personally, I’m switching more often to Codex 5.4-medium than Opus or Sonnet Feels like we’re approaching a plateau. And when that happens, the shift becomes obvious: LLMs won’t replace software engineers — they’ll expose how much real engineering still matters. Because getting to the final outcome isn’t just about generating code. It’s about making it work… even when the model doesn’t

Elliot Arledge @elliotarledge

2 months ago

x.com/i/article/2041…

23 30 351 398K 406

The real fear isn’t that AI can write code. It’s what that implies. If AI can code, maybe the job can be replaced. So people resist it — not because it’s wrong, but because it’s uncomfortable. But writing code was never the job. It was just the visible part. The real work is: - understanding messy problems - making decisions with incomplete context - owning outcomes Code is just how that work shows up. AI doesn’t remove the job. It removes the illusion that typing code was the job. The shift is simple, not easy: “I write code” → “I build things that work.” Those who make that shift gain leverage. Those who don’t fight the wrong battle.

Over the last 2–3 weeks, I've been reaching for GPT-5.4 more than Claude Opus 4.6. Mostly for feature planning and code reasoning. Opus 4.6 - Occasionally misses important context in the codebase → Overlooks critical details — even in code it wrote itself → Plans look fine on the surface → Gaps show up when you go one layer deeper GPT-5.4 - Plans are more complete and better structured → Connects context across files → Catches edge cases Opus skips → Feels more reliable on system-level changes The interesting part: - I cross-check GPT's plan with Opus - Opus now often agrees with it - Sometimes reinforces the exact same approach - That wasn't happening a few weeks ago Feels like a subtle shift in how these models behave under real coding workloads. For planning and architecture work, GPT-5.4 is my default now.