It is the question every IT manager at a large company asks: "will this system still work when we have 100,000+ items?" The short answer: if the system was not built for Big Data, no. This article explains how Hunt ERP handles 300,000+ SKUs with response times measured in seconds.
The challenge: why ordinary systems slow to a crawl
With 300,000 items, your products table holds 300,000 rows. A simple search like "find the product called red" scans all 300,000 of them, one at a time. Without optimisation, that search takes minutes.
Now picture:
- 300,000 items
- 1,000,000 invoices a year
- 50 staff searching the system simultaneously
An ordinary system chokes. The server falls over. Your team waits 30 seconds for every click.
The Big Data architecture inside Hunt ERP
1. Smart indexes
Every column that gets searched repeatedly carries an index. In Hunt ERP, searching 300,000 items takes milliseconds rather than seconds.
-- example of a composite index on the products table INDEX idx_search ON products(name, barcode, code, brand_id, category_id); INDEX idx_active_products ON products(is_active, deleted_at);
2. Multi-layer caching
| Layer | Technology | Latency |
|---|---|---|
| Object cache | Memcached/Redis | ~0.1ms |
| Query cache | Redis | ~1ms |
| Database cache | MySQL InnoDB buffer | ~5ms |
| Disk | NVMe SSD | ~50ms |
Hunt ERP layers three caches in sequence. Repeated requests are served from cache, never from the database.
3. Read replicas
Writes go to the master database. Reads — reporting, search — go to replicas. That spreads the load and keeps search fast even at peak hours.
4. ZLIB data compression
Invoice data and attachments are ZLIB-compressed, cutting storage by 60-70% and making disk I/O considerably faster.
5. Smart pagination
Rather than trying to render 300,000 products on one page (which is impossible), the system serves 50 per page with fast navigation — and it uses cursor-based pagination rather than offset, which is around 100x quicker at depth.
6. Lazy loading for images
Images do not all load at once. They load as the user scrolls to them, which saves roughly 90% of the bandwidth.
7. Reporting across millions of rows
A report like "top 100 products sold this year" runs across millions of sales lines. Hunt ERP handles it with:
- Materialised views refreshed hourly
- Pre-computed aggregate tables
- Background jobs for the heavy reports
The real benchmark: 300,000 items
| Operation | Time |
|---|---|
| Search a product by name | 0.3 seconds |
| Load the product catalogue (page 1, 50 items) | 0.5 seconds |
| Add a new product | 0.4 seconds |
| Issue an invoice with 50 line items | 1.2 seconds |
| Best-sellers report (full year) | 2.1 seconds |
| Full inventory report (300K products) | 4.5 seconds |
These are real numbers, measured on a live customer running a 300,000+ item catalogue.
Who actually needs a Big Data ERP?
- Office furniture companies with model/size/colour variants (thousands of SKUs)
- Chemical and pharmaceutical manufacturers (thousands of compounds)
- Retail chains (stores carrying 10,000+ products)
- Distributors with extended catalogues
- Large e-commerce operations
- Bookshops and stationers with very deep catalogues
Migrating your data to Hunt ERP
- Proof of concept: load a 10,000-product sample and benchmark it
- Migration: our team moves your data off the old system
- Validation: full testing against the complete dataset
- Go-live: launch with a month of intensive support
The bottom line
If your business carries a very large catalogue, Hunt ERP is built for it. Tested in production at 300,000+ items and still responding in seconds. Big Data architecture from the ground up, intelligent caching, read replicas, and live reporting across millions of rows.
Book a live demo and our team will show you the system running on data the size of yours.
Ready to see your company reports done right? Request a custom quote or start with a free trial .