Every trading session, Stoxmine ingests, processes, scores, and delivers intelligence on 5,000+ stocks in under 90 seconds. Here's how.
The Data Pipeline
Our pipeline runs in 4 stages, each optimised for throughput and latency:
Stage 1: Ingestion (0–15 seconds)
We pull data from multiple sources simultaneously:
- Price & volume: Real-time tick data from NSE & BSE via licensed API feeds
- Corporate data: Quarterly results, shareholding patterns, corporate actions from exchange filings
- Institutional flows: Bulk/block deal data, FII/DII segment-wise data from NSDL & CDSL
- News: RSS feeds and API integrations from 50+ Indian financial media outlets
Total daily ingestion: 2.8 million+ data points across all sources.
Stage 2: Normalisation & Feature Engineering (15–45 seconds)
Raw data is cleaned, normalised, and transformed into 127 quantitative features per stock. This includes:
- Z-score normalisation for cross-sector comparison
- Rolling window calculations (5, 10, 20, 50, 200-day)
- Relative strength vs. sector and market benchmarks
- NLP sentiment scoring on news headlines
- Chart pattern recognition via rule-based + ML hybrid models
Stage 3: Scoring Engine (45–75 seconds)
The 127 features feed into our 5-pillar weighted scoring model. Each pillar produces a sub-score (0–100), which are combined using dynamically adjusted weights based on the current market regime (trending, range-bound, or volatile).
The output: one Confidence Score per stock, updated every trading session.
Stage 4: Delivery & Alerting (75–90 seconds)
Scores are pushed to our API, cached at the edge via Vercel, and delivered to users. The alert engine simultaneously scans for:
- Score band changes (e.g., a stock moving from Hold to Buy)
- Golden crosses and death crosses
- Volume surge anomalies (>2x 20-day average)
- Bulk/block deal detections
- Earnings surprise triggers
Infrastructure Stack
- Compute: Python-based scoring engine on AWS Lambda (serverless, auto-scaling)
- Database: PostgreSQL (TimescaleDB extension for time-series) + Redis for hot caches
- API: FastAPI with async endpoints, <50ms p99 latency
- Frontend: Next.js 15 on Vercel Edge — ISR for scores, SSR for stock detail pages
- ML pipeline: scikit-learn + XGBoost for pattern recognition; PyTorch for experimental sentiment models
- Monitoring: Grafana + Prometheus, PagerDuty for on-call
Why Speed Matters
In equity markets, stale data is dangerous data. A score computed 2 hours ago doesn't reflect a breaking news event or a sudden FII sell-off. Our 90-second end-to-end pipeline ensures that when you open Stoxmine, you're seeing the most current intelligence available — not yesterday's analysis.
We're working to bring this down to under 30 seconds with streaming ingestion — stay tuned.
Ready to see Stoxmine in action?
Join 12,000+ investors who get institutional-grade intelligence — before the crowd.