Vision to LLM, in production

Context

Two employers, one continuous problem: getting models out of a notebook and into something that runs unattended, at scale, for people who do not know a model is involved.

At Verizon that meant computer vision. I trained and fine-tuned custom detectors and put them behind a real-time video pipeline on AWS for a smart security product with anomaly detection and user alerting, built the ingestion and processing pipelines for IoT telemetry and dashcam imagery that fed driver safety analytics, and delivered embedded CV applications in Python and C++ that ran inference on the devices themselves. Near the end I also built one of the team's first production LLM projects — a full-stack chatbot using retrieval-augmented generation.

At Swiss Re it meant language and data at a different kind of scale: large-scale processing and entity extraction pipelines in Palantir Foundry over financial data, and LLM-powered internal tools for investment analysis, including agent-based systems for the messier use cases.

I also spent a year as Scrum Master alongside the development work, which mostly taught me how much of engineering is deciding what not to build.

Constraints

Real time means real time. A security product that alerts thirty seconds late is not a late security product, it is a different and much worse product. That single requirement propagates backwards through every stage of the pipeline.

The edge is not the cloud. A detector that runs comfortably on a GPU instance has to fit a dashcam's compute, memory, and thermal budget — and a vehicle is a place where you cannot assume the connection you need is the connection you have.

Financial data punishes silence. Throughput mattered, but robustness and traceability mattered more. An extraction pipeline that fails loudly costs an afternoon; one that is quietly wrong costs the confidence of everyone downstream who has already acted on it.

Architecture and decisions

Custom-trained detectors rather than off-the-shelf weights. The domain gap is the whole problem. General-purpose checkpoints are trained on well-lit, well-framed, centre-weighted images; a security camera at night and a windscreen-mounted camera in rain are neither. Training on the actual data distribution let me use a smaller backbone than a general model would have needed, which then paid for itself again in latency.

Splitting inference between cloud and device by asking what the bottleneck is. Where the constraint was model capacity, inference belonged in the AWS pipeline. Where the constraint was bandwidth — dashcams generate far more video than you would ever want to ship — inference belonged on the device in C++, and only the events travelled. This is the decision I would defend most strongly: it is not a preference for edge or cloud, it is a per-workload answer to which resource is actually scarce.

Decoupling ingest from inference. The ingestion pipelines for telemetry and imagery were built so that changing a model did not mean touching ingest. Models change far more often than data contracts do, and coupling them means every retrain becomes a deployment risk.

Tuning alerting for precision over recall, past a point. The instinct in anomaly detection is to catch everything. But a user who has been woken three times by nothing stops trusting the system, and a system that is not trusted has zero effective recall regardless of its measured recall. Where to sit on that curve was a product decision informed by the model, not a threshold chosen from a validation set.

RAG before agents. The first production LLM project was retrieval-augmented rather than agentic, and that was the right order. Retrieval solved the problem that actually blocked adoption — answers grounded in real documents that a sceptical user could go and check — without introducing a system whose behaviour was hard to predict. Agents came later, at Swiss Re, for genuinely branching multi-step financial analysis where a fixed pipeline could not express the task.

Validation as part of the extraction pipeline, not a step after it. Given the failure mode above, entity extraction in Foundry was built so that low-confidence output surfaces as low-confidence rather than being silently emitted alongside everything else.

Outcome

Production systems in both domains: a real-time video pipeline with anomaly detection and user alerting, embedded CV applications running on-device, AWS ingestion pipelines feeding driver safety analytics, a production RAG chatbot, and LLM agent tooling for investment analysis in Foundry.

What I would do differently: on the LLM work, I would build the evaluation harness first. With computer vision the feedback loop is honest almost by accident — you have a validation set, mAP is a number, and a bad model looks bad. Generative systems will produce fluent, plausible, wrong output indefinitely and never once look broken, and "it seemed fine when I tried it" is not a measurement. Every LLM project I have worked on would have moved faster with evaluation in place from the first week, and that is now the first thing I build.