Blog
Optimize your company's online presence with our blog specializing in web development and valuable insights into the latest trends.
losangulo riscas pontos cruz

How to automate a product catalog without compromising performance

In high-performance ecommerce development, we believe true engineering isn't about what the user sees, but rather the robustness of data exchange between systems. We recently put this philosophy to the test by taking on the challenge of automatically synchronizing thousands of products from multiple external suppliers for one of our clients, Heavands. The scenario was complex: unstable REST APIs with significant performance bottlenecks, which needed to be transformed into a fluid, reliable operation.

Our solution did not involve adapting our platform to third-party plugins. Pelo contrário, we evolved our proprietary CMS to manage this complexity internally.

The Challenge: Data Inconsistency and Performance Risks

Integrating supplier ERPs with beta-version APIs presented real technical obstacles:

  • API Instability: We faced slow response times (averaging 3s per request), timeouts frequent and undocumented changes in data structure (JSON/XML).

  • Volumetric Data: We knew that real-time processing of +10,000 references during user navigation was unfeasible, as it would cause severe degradation of site performance.

The Solution: Asynchronous Architecture in our CMS

To ensure reliability and speed for Heavands, we developed an integration layer based on solid technical pillars within our platform:

1. Fallback Mechanisms and Intelligent Caching

To mitigate the instability of beta APIs, we implemented a robust fallback logic. If a call to a supplier fails or returns a server error, the system does not interrupt the process or display errors to the end customer. Instead, our CMS utilizes the last validated version of the data in cache, keeping the catalog functional while attempting reconnection in the background with an exponential backoff time.

2. Asynchronous Processing (Background Jobs)

We abandoned synchronous synchronization. We developed a system of asynchronous processing queues. Stock and price updates are placed in a queue and processed in the background (background jobs via CLI) by the server. This ensures that the end user's browsing experience is always fluid and fast, regardless of the integration workload.

This system allows us to process over 500 products per minute without impacting the front-end response time (Time to First Byte).

3. Data Normalization and Validation

Regardless of the format or structure each supplier sends, our CMS normalizes all information into a unified database structure.

  • Example: One supplier sends stock as "Yes/No", another as "1-5", "6-10", and a third as an exact number. Our validation layer translates everything into a standard integer numerical value before persisting it in the database.

4. Timezone Management and Logs

An unsuspected challenge was dealing with suppliers in different timezones. To ensure prices and availability were correct, we normalized all timestamps to UTC in our database, converting them only upon display to the final user. Furthermore, we implemented an alert system based on structured logs. If an API fails three consecutive times, our technical team receives an automatic alert with the error code, allowing us to act proactively before the impact is visible to the user.

Technical Insight and Success Metrics

The technical value of such a project lies not only in connectivity but in our ability to manage uncertainty. True engineering isn't about building a system that works under ideal conditions; it is about creating a resilient ecosystem that maintains data integrity and store performance even when external sources fail.

We reduced the product view error rate from 15% (with direct APIs) to <0.1% with our abstraction layer.

Conclusion: Stability as a Competitive Advantage

By developing these integration solutions directly into the core of our proprietary CMS, we are not just connecting systems; we are creating an abstraction layer that protects the business. We ensure that the stability of the online store is not held hostage by the technical inconsistency of third parties, which is much more common than one might think. Our ability to manage the uncertainty of external data is what ensures a seamless user experience and catalog integrity, regardless of failures in supplier APIs.

In our opinion, the success of this project demonstrates that investing in custom technology is the safest path for companies aiming for digital leadership and operational efficiency. By eliminating human error and manual work in catalog management, and by ensuring reliable, asynchronous stock and price synchronization, we allow teams to focus on what really matters: strategic business growth and customer satisfaction.

Other Articles
Back
Contacte-nos Contacte-nos