Abstract Data Source Support in LogExpert

[TheNicker]

Proposal: Abstract Data Source Support in LogExpert

Summary

Currently, LogExpert operates solely on the concept of reading from file(s), effectively treating the file system as the only log data source. While this works for many use cases, modern logging environments increasingly rely on diverse, non-file-based sources. To make LogExpert more extensible and adaptable, it is proposed to introduce an abstraction for data sources.

Motivation

A file is just one of many possible representations of a log data stream. Treating it as a special case of a more generic IDataSource concept would allow LogExpert to support a broader range of logging inputs, including but not limited to:

1. File system (current implementation)
2. gRPC endpoints
3. Syslog streams (via network)
4. ETW (Event Tracing for Windows)
5. Custom user-defined sources

This flexibility would significantly enhance LogExpert's usability in enterprise, cloud-native, and developer tooling contexts.

Proposed Implementation (Draft – Subject to Review)

To enable this, the following technical approach is recommended:

• Refactor LogFileReader.cs to extract an interface, tentatively named IDataSource.
• Define IDataSource as a general contract for retrieving and streaming log entries.
• Create concrete implementations, including:
  • FileDataSource (based on existing logic)
  • GrpcDataSource
  • SyslogDataSource
  • EtwDataSource
  • CustomDataSource (for end-user extensibility)
• Ensure backward compatibility by treating the file-based implementation as the default.

This is a preliminary design and should be reviewed for architectural soundness, API usability, and integration considerations.

Benefits

• Increased flexibility and extensibility
• Support for real-time and remote logging sources
• Improved adoption in diverse environments
• Lower barrier for third-party contributions

Next Steps

• Review and finalize the IDataSource interface design
• Modularize log reading logic in the current codebase
• Prototype support for at least one new data source (e.g., gRPC or syslog)
• Document plugin API and usage patterns for community involvement

[Hirogen]

This should already be supported to a certain extend, if you look at the sftpplugin, it kind of does that.

ILogFileInfo is basically the IDataSourcel, as long as you return a stream it should work. LogfileReader maybe needs to support different kinds of streams.