Public: Shopify Sidekick merchant assistant

Architectural notes

Per Shopify Engineering's "Building production-ready agentic systems"
(ICML 2025 talk by McNamara, Lafferty, Garner) and the RailsWorld 2025 talk,
Sidekick is a merchant assistant built on Ruby on Rails with an in-house agentic
loop; no third-party agent framework is named.

Tool calling scaled from a simple system to 50+ tools using Just-in-Time (JIT)
instructions delivered alongside tool data instead of bloating the system prompt.

Evaluation uses multiple custom LLM-as-judge models calibrated to human
annotators (Cohen's Kappa ~0.61 vs a 0.69 human baseline) plus an LLM merchant
simulator (in-house, represented as eval: local).

Shopify fine-tunes the model with GRPO (Group Relative Policy Optimization) and
an N-stage gated reward system; this is RL training, not a catalog component.

Not publicly disclosed in these sources: the base LLM model/provider, deployment
platform, observability tooling, or retrieval/memory systems.