At the heart of any observability platform lies the ability to handle massive volumes of logs efficiently, both in terms of cost and compliance. With Kloudfuse 3.0, we’re addressing two critical challenges faced by enterprise customers—log retention costs and efficient retrieval—through the introduction of Archival and Hydration features.

The Problem: Compliance and Cost in Log Management

Long-term log retention is essential for enterprises, especially for audit and legal compliance in financial services, healthcare, and other regulated industries. 

Traditional approaches to storing logs in real-time observability platforms often come with high storage costs and limited retention periods. For many organizations, this poses a challenge when compliance requirements demand longer retention periods while also keeping costs in check.

In addition to compliance, organizations need flexibility in how they store and retrieve logs. Logs can quickly balloon in size. Storing vast amounts of logs without proper tagging, filtering, and searching capabilities makes retrieval inefficient and costly.

Our Solution: Archival and Hydration

In Kloudfuse 3.0, Archival enables users to store logs in an object store of their choice (e.g., S3). The logs are stored in an easy-to-read, JSON format and are optimally compressed to minimize storage costs. With this feature, customers can manage their compliance requirements by keeping logs for extended periods without burdening their active storage systems. Logs are stored by timestamp, allowing customers to access data at a granular level that is easy to navigate.

Complementing Archival is the Hydration feature, which allows users to selectively retrieve archived logs back into the Kloudfuse platform. Users can apply filters such as time ranges or specific tags to only hydrate logs that are relevant to their search, ensuring that they don't waste time or resources retrieving unnecessary data. 

For instance, if a user needs to review logs from six months ago for an audit, they can apply filters and only hydrate the logs that are relevant to their specific audit period. This prevents the need to load terabytes of data unnecessarily, reducing time and operational overhead.

Together, these features simplify log management by decoupling storage from retrieval, meeting compliance needs with extended retention and cutting operational costs through efficient archiving.

Bonus Features of Kloudfuse: Tagging & Search

Kloudfuse adds value through LogParser, which tags logs during hydration, enabling efficient classification and future searches. This also allows internal teams to track and manage log usage for each department, adding an extra layer of cost control for large enterprises.

Additionally, users can Search across both current and previously archived logs—using tags and facets extracted during log fingerprinting—once archived logs are hydrated back to the system. This provides seamless access to logs through Facet Analytics, regardless of whether they’re archived or active.

Challenges and Rewards in Building the Feature

From an engineering perspective, building this capability presented several challenges. One of the key tools we used was Vector, an open-source log processing tool. A key lesson learned is that transitioning from fluent-bit, which limited us with performance and only supported gzip compression (resulting in higher costs for customers due to poor compression), to vector allowed us to overcome these issues and achieve both optimal performance and the desired compression format, enhancing overall customer value.

Log Archival and Hydration evolved significantly from the initial concept to its final form. Navigating that uncertainty and building a solution that helps our customers move from proof-of-concept (POC) into paying clients has been particularly gratifying.

Conclusion

Kloudfuse 3.0’s Archival and Hydration features offer a powerful solution for enterprises looking to balance the need for long-term log retention with the operational cost of storing and retrieving those logs. By leveraging object stores for archival and providing selective hydration capabilities, Kloudfuse enables customers to meet their compliance needs while optimizing storage costs. 

This feature is already proving valuable for some of our new customers, and we expect to see broader adoption as more organizations recognize the benefits of this approach.