We have created a new query language to overcome the limitations of existing solutions in the logging. In this blog, we’ll dive into the technical specifics, the challenges we faced with existing technologies like LogQL, and the motivation behind our new query language—fuseQL.

Until now, we have supported LogQL as our query language for log analysis. While we uphold an open architecture and open standards, many of our customers transitioning from platforms like Sumo Logic and Splunk require more advanced logging query capabilities. With the release of fuseQL, we now offer a comprehensive set of operators that empower users to execute complex queries and aggregations, essential for scalable and in-depth data analysis of enterprise workloads.

The Limitations of LogQL

LogQL, the query language associated with Grafana's Loki, has been a popular choice for many users; however, it presents several critical limitations that hinder its effectiveness for advanced querying needs:

• Limited Query Language: LogQL is less expressive compared to other query languages, with limited filtering and aggregation functions, which can restrict complex queries and data analysis.

• Performance Concerns: Queries can become slower with large datasets in high-cardinality logs (logs with many unique identifiers), leading to performance bottlenecks during log analysis. 

• Licensing Constraints: Governed by the AGPL license, any modifications made to LogQL must be shared with the community, which may not align with the preferences of all our customers.

These constraints highlight the need for a more versatile query language that can better serve users' diverse needs in logging and data analysis.

Designing fuseQL: An Engineering Perspective

The development of FuseQL has been an iterative process, rooted in our engineering expertise and informed by the shortcomings we identified in existing solutions. Our goal was to create a query language that provides the functionality needed for modern data analysis, including advanced filtering, aggregation, and a comprehensive set of operations.

Key Features of fuseQL

1. Rich Set of Operators: fuseQL introduces a comprehensive set of operators that enable complex queries. Users can perform multi-dimensional aggregations and filters, providing greater analytical flexibility.

2. High-Performance Full-Text Search: With our indexing system that catalogs both log content and metadata, fuseQL ensures rapid analysis, even in high cardinality and high-volume log environments.

3. Advanced Functions: Because FuseQL is richer in aggregation, parsing, and computational complexity, it supports advanced functions such as anomaly detection or outlier identification directly from log data, empowering users to extract more value from their logs. 

4. Smarter Alerts: Another advantage of FuseQL is its ability to create smarter alerts. With more expressive operators, users can define complex conditions and thresholds tailored to specific use cases, detecting not only simple errors but also patterns indicating potential issues or trends over time. As a result, users can proactively address problems before they escalate. 

5. Dual Data Representation: fuseQL supports both time series (like logQL) and relational data models, allowing users to harness the strengths of each format for complex analyses without data representation constraints.

6. Compatibility with LogQL: For users familiar with LogQL, fuseQL offers compatibility, easing the transition and maintaining access to both languages.

Vinay’s Perspective:

Building a new query language is inherently complex. Our process paralleled compiler design, requiring critical architectural decisions around language grammar and execution strategies. Each iteration has brought us closer to a cohesive and high-performing product that meets our users' needs while ensuring compatibility with existing systems like LogQL and Loki.

Sushant’s Insights:

With a background in data science and machine learning, I’ve focused on advancing Kloudfuse’s AI capabilities. By incorporating fuseQL into our platform, we enable our machine learning and statistical models to extract deeper insights from logs through correlations, forecasts, and confidence scores.

Ashvin’s Take:

As a recent computer science graduate, I found it rewarding to implement and optimize mathematical operators and functions in FuseQL. Enhancing the efficiency of these operations has significantly improved our system’s performance, directly benefiting our customers. Collaborating with the dynamic Kloudfuse team has been an incredible learning experience.

Conclusion

As we continue to refine FuseQL, we are excited about its potential to revolutionize log analysis. By overcoming the limitations of LogQL, we aim to deliver a powerful, flexible query language that empowers users to unlock the full potential of their log data.

Stay tuned for updates as we progress towards the full release of FuseQL!