masterlog/README.md

# MasterLog - Logging Library for Golang

A powerful, extensible, and zero-dependency logging library for Go with built-in support for structured logging, colored output, and data pseudonymization.

## Features

- 🎨 **Colored Console Output** - Beautiful, syntax-highlighted logs with automatic terminal detection
- 📝 **Structured Logging** - Support for custom fields and structured data
- 🔒 **Data Pseudonymization** - Built-in HMAC-based deterministic pseudonymization for sensitive data
- 📊 **Multiple Encoders** - Formatted text and JSON encoding out of the box
- 📁 **Flexible Writers** - Console and file writers with easy extensibility
- 🎯 **Zero Dependencies** - Uses only Go standard library
- 🔧 **Extensible Architecture** - Easy to add custom encoders and writers
- ⚡ **Performance** - Efficient logging with minimal overhead
- 🎛️ **Configurable Log Levels** - Trace, Debug, Info, Warn, Error

## Installation

```bash
go get git.secnex.io/secnex/masterlog
```

## Quick Start

### Basic Usage

```go
package main

import (
    "git.secnex.io/secnex/masterlog"
)

func main() {
    // Simple logging using the global logger
    masterlog.Info("Hello, World!")
    
    // Logging with fields
    masterlog.Info("User logged in", map[string]interface{}{
        "user_id": 12345,
        "ip":      "192.168.1.1",
    })
}
```

**Output:**
```
2025-11-10T05:06:02+01:00 INF main.go:9 > Hello, World! go_version=go1.25.3 pid=12345
2025-11-10T05:06:02+01:00 INF main.go:12 > User logged in user_id=12345 ip=192.168.1.1 go_version=go1.25.3 pid=12345
```

### Configuring the Global Logger

Configure the global logger once at application startup:

```go
package main

import (
    "git.secnex.io/secnex/masterlog"
)

func main() {
    // Configure global logger at startup
    masterlog.SetLevel(masterlog.LevelDebug)
    
    // Add file writer
    fileWriter, _ := masterlog.NewFileWriter("app.log")
    defer fileWriter.Close()
    masterlog.AddWriter(fileWriter)
    
    // Add JSON encoder
    masterlog.AddEncoder(&masterlog.JSONEncoder{})
    
    // Configure pseudonymization
    pseudonymizer := masterlog.NewPseudonymizerFromEnv("LOG_SECRET")
    masterlog.SetPseudonymizer(pseudonymizer)
    masterlog.AddSensitiveFields("user_id", "email", "ip")
    
    // Now all package-level functions use the configured global logger
    masterlog.Info("Application started")
    masterlog.Debug("Debug message")
}
```

**Benefits:**
- Configure once, use everywhere
- No need to pass logger instances around
- Simple and convenient for most use cases

## Table of Contents

- [Log Levels](#log-levels)
- [Structured Logging](#structured-logging)
- [Colored Output](#colored-output)
- [Encoders](#encoders)
- [Writers](#writers)
- [Custom Logger Instances](#custom-logger-instances)
- [Data Pseudonymization](#data-pseudonymization)
- [Extending MasterLog](#extending-masterlog)
- [API Reference](#api-reference)
- [Best Practices](#best-practices)

## Log Levels

MasterLog supports five log levels:

- `TRC` (Trace) - Most verbose, for detailed debugging
- `DBG` (Debug) - Debug information
- `INF` (Info) - General informational messages
- `WRN` (Warn) - Warning messages
- `ERR` (Error) - Error messages

### Setting Log Level

```go
// Set level for default logger
masterlog.SetLevel(masterlog.LevelDebug)

// Create logger with specific level
logger := masterlog.New(masterlog.LevelTrace)
logger.Trace("This will be logged")
logger.Debug("This will also be logged")
```

### Logging Methods

```go
masterlog.Trace("trace message")
masterlog.Debug("debug message")
masterlog.Info("info message")
masterlog.Warn("warning message")
masterlog.Error("error message")
```

## Structured Logging

MasterLog supports structured logging with custom fields. Custom fields are displayed before default fields (go_version, pid).

```go
masterlog.Info("User action", map[string]interface{}{
    "user_id": 12345,
    "action":  "purchase",
    "amount":  99.99,
    "ip":      "192.168.1.1",
})
```

**Output:**
```
2025-11-10T05:06:02+01:00 INF main.go:12 > User action user_id=12345 action=purchase amount=99.99 ip=192.168.1.1 go_version=go1.25.3 pid=12345
```

### Field Ordering

Custom fields (user-provided) are always displayed before default fields:
1. Custom fields (user-provided)
2. Default fields (go_version, pid)

## Colored Output

MasterLog automatically detects if output is going to a terminal and applies colors accordingly. Colors are never written to files.

### Color Scheme

- **Timestamp**: Dark gray
- **Log Level**: 
  - `TRC`: Gray
  - `DBG`: Cyan
  - `INF`: Green
  - `WRN`: Yellow
  - `ERR`: Red
- **File:Line**: Dark gray
- **Message**: White
- **Field Keys**: Turquoise
- **Field Values**: White

### Example Output

When logging to a terminal, you'll see beautifully colored output. When redirecting to a file or non-terminal, colors are automatically disabled.

```go
masterlog.Info("Colored output example", map[string]interface{}{
    "key": "value",
})
```

## Encoders

Encoders determine the format of log entries. MasterLog provides two built-in encoders:

### FormattedEncoder

The default encoder that produces human-readable formatted text.

```go
logger := masterlog.New()
// FormattedEncoder is used by default
logger.Info("Formatted log")
```

**Output:**
```
2025-11-10T05:06:02+01:00 INF main.go:12 > Formatted log go_version=go1.25.3 pid=12345
```

### JSONEncoder

Produces JSON-formatted log entries, perfect for log aggregation systems.

```go
logger := masterlog.New()
logger.AddEncoder(&masterlog.JSONEncoder{})
logger.Info("JSON log", map[string]interface{}{
    "key": "value",
})
```

**Output:**
```json
{"timestamp":"2025-11-10T05:06:02+01:00","level":"INF","file":"main.go","line":12,"message":"JSON log","fields":{"key":"value","go_version":"go1.25.3","pid":12345}}
```

**Note:** JSON encoder output is automatically excluded from console output when using the default logger to keep console logs readable.

### Multiple Encoders

You can use multiple encoders simultaneously:

```go
logger := masterlog.New()
logger.AddEncoder(&masterlog.JSONEncoder{})
// Now logs will be written in both formatted and JSON formats
logger.Info("Dual format log")
```

## Writers

Writers determine where log entries are written. MasterLog provides two built-in writers:

### ConsoleWriter

Writes to stdout (default). Automatically detects if output is a terminal.

```go
logger := masterlog.New()
// ConsoleWriter is used by default
logger.Info("Console log")
```

### FileWriter

Writes log entries to a file.

**Using with global logger (recommended):**

```go
// Add file writer to global logger
fileWriter, err := masterlog.NewFileWriter("app.log")
if err != nil {
    log.Fatal(err)
}
defer fileWriter.Close()

masterlog.AddWriter(fileWriter)
masterlog.Info("File log")  // Uses global logger
```

**Using with custom logger:**

```go
logger := masterlog.New()
fileWriter, err := masterlog.NewFileWriter("app.log")
if err != nil {
    log.Fatal(err)
}
defer fileWriter.Close()

logger.AddWriter(fileWriter)
logger.Info("File log")
```

### Multiple Writers

You can write to multiple destinations simultaneously:

```go
logger := masterlog.New()

// Add file writer
fileWriter, _ := masterlog.NewFileWriter("app.log")
defer fileWriter.Close()
logger.AddWriter(fileWriter)

// Logs will be written to both console and file
logger.Info("Multi-destination log")
```

## Global Logger vs Custom Instances

MasterLog provides two ways to use logging:

### Global Logger (Recommended for Most Cases)

The global logger is pre-configured and ready to use. Configure it once at startup:

```go
func main() {
    // Configure global logger at startup
    masterlog.SetLevel(masterlog.LevelDebug)
    
    // Add file writer
    fileWriter, _ := masterlog.NewFileWriter("app.log")
    defer fileWriter.Close()
    masterlog.AddWriter(fileWriter)
    
    // Add JSON encoder
    masterlog.AddEncoder(&masterlog.JSONEncoder{})
    
    // Use package-level functions anywhere in your code
    masterlog.Info("This uses the global logger")
    masterlog.Debug("No need to pass logger instances around")
}
```

**Advantages:**
- Simple and convenient
- No need to pass logger instances
- Configure once, use everywhere
- Perfect for most applications

### Custom Logger Instances

Create custom logger instances when you need different configurations for different components:

```go
// Create logger with custom level
logger := masterlog.New(masterlog.LevelDebug)

// Add JSON encoder
logger.AddEncoder(&masterlog.JSONEncoder{})

// Add file writer
fileWriter, _ := masterlog.NewFileWriter("custom.log")
defer fileWriter.Close()
logger.AddWriter(fileWriter)

logger.Info("Custom logger example")
```

**Use Cases:**
- Different log levels for different components
- Separate log files for different modules
- Different encoders for different outputs
- Component-specific pseudonymization rules

### Replacing the Global Logger

You can also replace the global logger with a custom instance:

```go
// Create and configure a custom logger
customLogger := masterlog.New(masterlog.LevelTrace)
customLogger.AddEncoder(&masterlog.JSONEncoder{})

// Replace the global logger
masterlog.SetDefaultLogger(customLogger)

// Now all package-level functions use your custom logger
masterlog.Info("Uses the custom logger")
```

### Getting the Global Logger

Access the global logger instance directly if needed:

```go
globalLogger := masterlog.GetDefaultLogger()
globalLogger.AddWriter(fileWriter)
```

## Data Pseudonymization

MasterLog includes built-in support for deterministic pseudonymization of sensitive data using HMAC-SHA256. This allows you to:

- Protect sensitive information in logs
- Maintain traceability (same input always produces same pseudonymized output)
- Comply with data protection regulations

### Basic Usage

```go
// Create pseudonymizer with secret
pseudonymizer := masterlog.NewPseudonymizerFromString("your-secret-key")
logger := masterlog.New()
logger.SetPseudonymizer(pseudonymizer)

// Mark fields as sensitive
logger.AddSensitiveFields("user_id", "email", "ip", "ssn")

// Log with sensitive data
logger.Info("User logged in", map[string]interface{}{
    "user_id": 12345,           // Will be pseudonymized
    "email":   "user@ex.com",   // Will be pseudonymized
    "ip":      "192.168.1.1",   // Will be pseudonymized
    "action":  "login",         // Not sensitive, remains unchanged
})
```

**Output:**
```
2025-11-10T05:06:02+01:00 INF main.go:15 > User logged in user_id=a1b2c3d4 email=e5f6g7h8 ip=i9j0k1l2 action=login go_version=go1.25.3 pid=12345
```

### Deterministic Behavior

The same input always produces the same pseudonymized output, allowing you to trace the same entity across multiple log entries:

```go
logger.Info("User action 1", map[string]interface{}{
    "user_id": 12345,  // Pseudonymized to: a1b2c3d4
})

logger.Info("User action 2", map[string]interface{}{
    "user_id": 12345,  // Pseudonymized to: a1b2c3d4 (same as above)
})
```

### Configuration Options

#### Using Environment Variables

For production, use environment variables to store the secret:

```go
// Read secret from environment variable
pseudonymizer := masterlog.NewPseudonymizerFromEnv("LOG_PSEUDONYMIZER_SECRET")
logger.SetPseudonymizer(pseudonymizer)
```

#### Custom Hash Length

Adjust the length of pseudonymized values (default: 8 hex characters):

```go
pseudonymizer := masterlog.NewPseudonymizerFromString("secret")
pseudonymizer.SetHashLength(16)  // Use 16 hex characters instead of 8
```

#### Managing Sensitive Fields

```go
// Add single field
logger.AddSensitiveField("user_id")

// Add multiple fields
logger.AddSensitiveFields("email", "ip", "ssn", "phone")

// Remove field (if needed)
pseudonymizer := masterlog.NewPseudonymizerFromString("secret")
pseudonymizer.RemoveSensitiveField("phone")
```

### Security Considerations

1. **Secret Management**: Store the pseudonymization secret securely (environment variables, secrets manager)
2. **Secret Rotation**: Changing the secret will produce different pseudonymized values
3. **Hash Length**: Longer hashes provide better uniqueness but are less readable
4. **Field Selection**: Only mark truly sensitive fields to maintain log usefulness

## Extending MasterLog

MasterLog is designed to be easily extensible. You can create custom encoders and writers.

### Custom Encoder

```go
type CustomEncoder struct{}

func (e *CustomEncoder) Encode(entry masterlog.Entry) ([]byte, error) {
    // Your custom encoding logic
    return []byte("custom format"), nil
}

// Usage
logger := masterlog.New()
logger.AddEncoder(&CustomEncoder{})
```

### Custom Writer

```go
type CustomWriter struct {
    // Your writer fields
}

func (w *CustomWriter) Write(data []byte) error {
    // Your custom writing logic
    return nil
}

func (w *CustomWriter) Close() error {
    // Cleanup logic
    return nil
}

func (w *CustomWriter) SupportsColors() bool {
    return false  // or true if your writer supports colors
}

// Usage
logger := masterlog.New()
logger.AddWriter(&CustomWriter{})
```

## API Reference

### Package-Level Functions

#### Logging Functions

```go
func Trace(message string, fields ...map[string]interface{})
func Debug(message string, fields ...map[string]interface{})
func Info(message string, fields ...map[string]interface{})
func Warn(message string, fields ...map[string]interface{})
func Error(message string, fields ...map[string]interface{})
```

#### Configuration Functions

```go
func SetLevel(level Level)
func AddWriter(writer Writer)
func AddEncoder(encoder Encoder)
func SetPseudonymizer(pseudonymizer *Pseudonymizer)
func AddSensitiveField(fieldName string)
func AddSensitiveFields(fieldNames ...string)
func SetDefaultLogger(logger *MasterLogger)
func GetDefaultLogger() *MasterLogger
```

### Logger Methods

```go
type MasterLogger struct {
    // ...
}

func New(level ...Level) *MasterLogger
func (l *MasterLogger) SetLevel(level Level)
func (l *MasterLogger) AddWriter(writer Writer)
func (l *MasterLogger) AddEncoder(encoder Encoder)
func (l *MasterLogger) SetPseudonymizer(pseudonymizer *Pseudonymizer)
func (l *MasterLogger) AddSensitiveField(fieldName string)
func (l *MasterLogger) AddSensitiveFields(fieldNames ...string)
func (l *MasterLogger) Trace(message string, fields ...map[string]interface{})
func (l *MasterLogger) Debug(message string, fields ...map[string]interface{})
func (l *MasterLogger) Info(message string, fields ...map[string]interface{})
func (l *MasterLogger) Warn(message string, fields ...map[string]interface{})
func (l *MasterLogger) Error(message string, fields ...map[string]interface{})
```

### Pseudonymizer

```go
type Pseudonymizer struct {
    // ...
}

func NewPseudonymizer(secret []byte) *Pseudonymizer
func NewPseudonymizerFromString(secret string) *Pseudonymizer
func NewPseudonymizerFromEnv(envVar string) *Pseudonymizer
func (p *Pseudonymizer) SetHashLength(length int) error
func (p *Pseudonymizer) AddSensitiveField(fieldName string)
func (p *Pseudonymizer) AddSensitiveFields(fieldNames ...string)
func (p *Pseudonymizer) RemoveSensitiveField(fieldName string)
func (p *Pseudonymizer) IsSensitive(fieldName string) bool
func (p *Pseudonymizer) Pseudonymize(value interface{}) string
func (p *Pseudonymizer) PseudonymizeFields(fields map[string]interface{}) map[string]interface{}
```

### Writers

```go
func NewConsoleWriter() *ConsoleWriter
func NewFileWriter(filename string) (*FileWriter, error)
```

### Encoders

```go
func NewFormattedEncoder(useColors bool) *FormattedEncoder
// JSONEncoder has no constructor, use &masterlog.JSONEncoder{}
```

## Best Practices

### 1. Use Structured Logging

Always use structured fields instead of string concatenation:

```go
// ❌ Bad
masterlog.Info(fmt.Sprintf("User %d logged in from %s", userID, ip))

// ✅ Good
masterlog.Info("User logged in", map[string]interface{}{
    "user_id": userID,
    "ip":      ip,
})
```

### 2. Set Appropriate Log Levels

Use log levels appropriately:

- `TRC`: Detailed trace information (usually disabled in production)
- `DBG`: Debug information (disabled in production)
- `INF`: General information about application flow
- `WRN`: Warning conditions that don't stop execution
- `ERR`: Error conditions that need attention

### 3. Pseudonymize Sensitive Data

Always pseudonymize sensitive information:

```go
logger.AddSensitiveFields("user_id", "email", "ip", "ssn", "credit_card")
```

### 4. Use Environment Variables for Secrets

Never hardcode secrets:

```go
// ❌ Bad
pseudonymizer := masterlog.NewPseudonymizerFromString("hardcoded-secret")

// ✅ Good
pseudonymizer := masterlog.NewPseudonymizerFromEnv("LOG_PSEUDONYMIZER_SECRET")
```

### 5. Separate Logs by Environment

Use different configurations for development and production:

```go
logger := masterlog.New()

if os.Getenv("ENV") == "production" {
    // Production: JSON to file, no colors
    logger.AddEncoder(&masterlog.JSONEncoder{})
    fileWriter, _ := masterlog.NewFileWriter("/var/log/app.log")
    logger.AddWriter(fileWriter)
    
    // Enable pseudonymization
    pseudonymizer := masterlog.NewPseudonymizerFromEnv("LOG_SECRET")
    logger.SetPseudonymizer(pseudonymizer)
    logger.AddSensitiveFields("user_id", "email", "ip")
} else {
    // Development: Formatted output with colors
    logger.SetLevel(masterlog.LevelDebug)
}
```

### 6. Close File Writers

Always close file writers when done:

```go
fileWriter, err := masterlog.NewFileWriter("app.log")
if err != nil {
    return err
}
defer fileWriter.Close()
```

### 7. Use Custom Logger Instances

Create separate logger instances for different components:

```go
var (
    apiLogger    = masterlog.New(masterlog.LevelInfo)
    dbLogger     = masterlog.New(masterlog.LevelDebug)
    authLogger   = masterlog.New(masterlog.LevelWarn)
)
```

## Examples

See the [example](example/) directory for complete working examples.

## License

This project is licensed under the MIT License - see the [LICENSE](LICENSE) file for details.

## Contributing

Contributions are welcome! Please feel free to submit a pull request.

## Support

Please feel free to open an issue if you have any questions or suggestions. You can also send an email to support@secnex.io.