Skip to content

Pillar VI — Technology Horizon

Scaling Architecture (Under 10K Users)

Shimmy scaling architecture diagram — data flow from client through moderation pipeline to feed generation

2.1 Stack design

1. Client & Edge Layer

  • Global User Clients: Users access the platform via Mobile Apps (iOS/Android) or tablets using standard secure protocols (HTTPS) for traditional requests and WebSockets (WS) for real-time updates (like feeds or notifications).
  • API Gateway: Acts as the single entry point for all client traffic. It handles:
  • Rate Limiting: Protects downstream services from being overwhelmed or attacked.
  • Authentication (JWT): Validates user identity tokens before routing traffic.
  • Request Routing: Directs user traffic either to the Core Social Services or Moderation Services.

2. Core Social Services & Primary Data Stores

This is the operational heart of the application where everyday user actions happen.

  • Core Social Services: Split into three decoupled microservices:
  • A. Research & Paper CRUD: Handles the creation, reading, updating, and deleting of academic papers. Because papers scale massively, this service talks to a Vitess-managed sharded MySQL cluster, split across multiple physical shards (SHARD_0, SHARD_1, SHARD_2) using a research_paper_id partitioning key.
  • B. Researcher Profiles & Feed Generation: Manages user feeds and timelines using native MySQL JOIN operations.
  • C. Identity & Relations: Manages user accounts, followers, and social graph connections, storing this data in a dedicated Identity, Profiles, & Relations MySQL database.
  • Redis Cluster: A high-speed, in-memory database used for global caching, fast timeline retrieval, user session data, and aggregating quick model votes.

3. The Event Ingestion & Asynchronous Queue

When a user uploads a paper or makes a post, it must be checked for toxicity or policy violations before going live globally.

  • POST_CREATED Event Ingestion: As soon as a post is made, a direct SQL insert injects a POST_CREATED event into a Moderation Queue.
  • MySQL SKIP LOCKED State: The queue uses a native MySQL concurrency feature (SELECT ... FOR UPDATE SKIP LOCKED). This allows multiple background worker processes to grab incoming posts simultaneously without locking the same row or stepping on each other's toes, making it an incredibly lightweight and fast volatile message queue.

4. Moderation Services (The AI & Human Pipeline)

This section uses an asymmetrical "Fast vs. Slow" design to balance speed with deep reasoning. Workers pull tasks asynchronously from the queue.

  • Tier 1: Fast Pass Workers (RoBERTa): Optimised for pure speed (<50ms processing time).
  • Uses a smaller language model (RoBERTa) to perform a rapid toxicity check. If a post is overwhelmingly clean, it passes immediately.
  • Tier 2: Slow Lane Workers (Gemma 3 Thinking Mode): Optimised for deep reasoning (>500ms processing time).
  • If Tier 1 is uncertain, or if the post requires complex contextual nuance, it gets routed here. It uses Gemma 3's advanced reasoning capabilities to cross-reference platform policies and multi-model context.
  • Human Mod Queue & Audit Logs: If the AI tiers cannot confidently make a decision, the event falls back to a dedicated MySQL queue for manual human review via Staff Clients. This also records all audit logs for tracking mod decisions.
  • Fast Model Cache (Redis): A shared cache layer that allows the workers and staff clients to quickly look up cached model results or flag states without hitting the primary databases.

Scaled Architecture (Growth Phase)

1. The Write Gateway & Fast Ingestion

When a user creates a post/shimmy (text, images, or video), it follows the blue path into the Write Gateway:

  • Asynchronous Trust Profiling (Triage Layer 0): Instead of choking the database with synchronous queries, the gateway intercepts the request and instantly checks a low-latency User Trust Cache (Redis). This tells the system if the poster is a trusted user, a known spammer, or a new account.
  • Optimistic Initial Write: The post is instantly dumped into ScyllaDB with a state of mod_status: pending_review. It is completely hidden from public feeds at this stage, freeing up the client connection in milliseconds.
  • The Event Backbone: Simultaneously, the write event is fired into the Kafka Event Stream to trigger asynchronous processing downstream.

2. Shifted-Left Moderation Pipeline (The Purple Box)

Shimmy Shield -our AI moderation system- reads from Kafka and uses a smart multi-tiered approach to maximise compute efficiency:

  • Early p-HASH Deduplication: Before burning expensive GPU cycles on vector embeddings, the media goes straight through a p-HASH Deduplication Cache. If the image/video matches a known piece of spam, violating content, or viral media, it is instantly routed without further ML inference.
  • The Multi-Tiered AI Gauntlet: Clean/unique items pass to the text normalizer and ViT Inference (for visual embeddings).
  • Triage Layer 2 (Fast Pass): Lightweight models (RoBERTa + Toxic-BERT) evaluate the text concurrently. If the safety score is clearly "clear" or "violation", it skips ahead to the persistence layer.
  • Triage Layer 3 (Slow Lane - GEMMA 3): If the score lands in the ambiguous "Grey Zone" (0.2−0.8), the system routes it to a heavy LLM (GEMMA 3) to break the tie using deep contextual awareness.

3. Decentralized Storage via CDC (The Red & Green Boxes)

To prevent database race conditions and the risky "dual-write" problem, we have introduced a robust CDC (Change Data Capture) Event Collector & Router:

  • The Triage Layer 3: Persistence Worker writes the final moderation verdict to the core PostgreSQL Engine (the source of truth for metadata, audit logs, and reports).
  • The CDC Collector & Router listens to database transaction logs. It seamlessly replicates and maps these state changes out to the rest of the Polyglot Data Layer:
  • Updates the mod_status to approved/hidden in ScyllaDB (which powers high-volume chronological feeds).
  • Streams the heavy Text & Media ViT embeddings into the Vector DB to serve as the structural framework for semantic search and discovery.

4. The Telemetry & Signal Feedback Loop

A recommendation system is only as good as its data. The bottom track captures real-time user behaviour:

  • The user scrolls through their app, generating implicit signals (e.g., how long they stare at a post via Dwell Time, likes, shares, or skips).
  • The Telemetry API ingests this massive stream, pumps it through Kafka, and passes it to an Analytical Pipeline.
  • This pipeline converts raw behaviour into mathematical vectors and instantly executes a Redis Update to mutate the user's Active User Preference Vectors inside the Redis Cluster.

5. The Recommendation & Read Engine (The Discovery Engine)

When the client requests a feed, the Feeds & Discovery Read Path (the orange/tan loop) jumps into action, combining the telemetry with data retrieval:

  • Step 1: Candidate Retrieval (ANN): The Personalised Feed Generation Service hits the Vector DB (which doubles as the Recommendation Candidate Pool). Using the user's Active User Preference Vectors from Redis, it performs an Approximate Nearest Neighbour (ANN) search to pull a few hundred posts that semantically match the user's real-time tastes.
  • Step 2: Blending with Momentum: The Read Gateway pulls from the Trending ZSets (Redis) to blend these highly personalised vector candidates with high-velocity, platform-wide viral content (Shimmy Momentum Scoring).
  • Step 3: Service Delivery: The final mixed, deduplicated, and ranked feed is delivered back to the Client Application with sub-second latency.