# 🧹 Temporary Files Cleanup Guide

## 📋 Overview

The `maintenance:cleanup-temp` command automatically removes old temporary files from designated directories in the storage system. This prevents storage buildup from test artifacts, temporary exports, CSV files, and other ephemeral data.

## 🎯 Objective

Delete obsolete temporary files safely, efficiently, and automatically to maintain optimal storage performance.

---

## 🔑 Key Concepts

### 1. **Retention Window**
- **Default value:** 24 hours
- **What it does:** Only deletes files older than this time
- **Why:** Allows active processes to complete and provides time for debugging
- **Configurable:** Via `--older-than` option (e.g., `24h`, `7d`)

### 2. **Safety Window**
- **Default value:** 10 minutes
- **What it does:** Protects recently modified files
- **Why:** Prevents deletion of files still being written or processed
- **How it works:** Skips any file modified within the last 10 minutes

### 3. **Exclusion Patterns**
- **Default patterns:** `*.lock`, `.gitignore`, `.placeholder`
- **What it does:** Never deletes files matching these patterns
- **Why:** Protects system files and lock files
- **Customizable:** Via `config/maintenance.php`

### 4. **Batch Processing**
- **Default value:** 500 files per batch
- **What it does:** Deletes files in small groups
- **Why:** Reduces filesystem pressure and allows for interruption
- **Configurable:** Via `--batch` option

### 5. **Cache Lock**
- **Key:** `maintenance:cleanup-temp`
- **Timeout:** 600 seconds (10 minutes)
- **What it does:** Prevents multiple instances from running simultaneously
- **Why:** Avoids conflicts and duplicate scanning

### 6. **Maximum Runtime**
- **Default value:** 120 seconds (2 minutes)
- **What it does:** Stops the command gracefully after this time
- **Why:** Prevents long-running processes; resumes on next scheduled run
- **Configurable:** Via `--max-runtime` option

### 7. **Cleanup Paths**
- **Default paths:**
  - `storage/app/temp` - General temporary files
  - `storage/framework/testing` - Test artifacts (SQLite databases, etc.)
- **Configurable:** Via `config/maintenance.php`

---

## 🚀 Command Usage

### Basic Syntax

```bash
php artisan maintenance:cleanup-temp [options]
```

### Available Options

| Option | Default | Description |
|--------|---------|-------------|
| `--dry-run` | false | Simulates execution without deleting files |
| `--path=` | config paths | Overrides and targets a specific directory |
| `--older-than=` | 24h | Custom retention period (e.g., `24h`, `7d`, `168`) |
| `--batch=` | 500 | Number of files to delete per batch |
| `--max-runtime=` | 120 | Maximum execution time in seconds |

---

## 📖 Usage Examples

### 1. Normal Execution (Production)

```bash
php artisan maintenance:cleanup-temp
```

**Result:** Deletes temporary files older than 24 hours from all configured paths.

**Expected output:**
```
🧹 Starting temporary files cleanup process...

🔄 Processing path: /path/to/storage/app/temp
  📅 Retention: 24 hours
  🛡️  Safety window: 10 minutes
  📝 Found 127 candidate files
  💾 Total size: 45.32 MB
  127/127 files deleted [████████████████████] 0.8s
  ✅ Deleted 127 files (45.32 MB)

🔄 Processing path: /path/to/storage/framework/testing
  📅 Retention: 24 hours
  🛡️  Safety window: 10 minutes
  📝 Found 16 candidate files
  💾 Total size: 256.00 KB
  16/16 files deleted [████████████████████] 0.2s
  ✅ Deleted 16 files (256.00 KB)
  🗑️  Cleaning empty directories...
  ✅ Removed 3 empty directories

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
📊 CLEANUP SUMMARY
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Paths processed:    2
  Files scanned:      143
  Files deleted:      143
  Space reclaimed:    45.58 MB
  Empty dirs removed: 3
  Duration:           1012 ms
  Errors:             0
  Mode:               PRODUCTION
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
✅ Cleanup completed successfully!
```

---

### 2. Simulation (Dry-Run)

```bash
php artisan maintenance:cleanup-temp --dry-run
```

**Result:** Shows what files would be deleted WITHOUT actually deleting them. **Use this first to verify.**

**Expected output:**
```
🧹 Starting temporary files cleanup process...
🔍 DRY-RUN MODE: No files will be deleted

🔄 Processing path: /path/to/storage/app/temp
  📅 Retention: 24 hours
  🛡️  Safety window: 10 minutes
  📝 Found 127 candidate files
  💾 Total size: 45.32 MB
  🔍 DRY-RUN: Would delete 127 files (45.32 MB)

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
📊 CLEANUP SUMMARY
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Files scanned:      127
  Files deleted:      0
  Space reclaimed:    45.32 MB
  Mode:               DRY-RUN (no files deleted)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
✅ Cleanup completed successfully!
```

---

### 3. Specific Directory

```bash
php artisan maintenance:cleanup-temp --path=storage/app/temp
```

**Result:** Only processes the specified directory, ignoring configured paths.

**Use case:** When you want to clean a specific location without affecting others.

---

### 4. Custom Retention Period

```bash
# Keep files for 7 days instead of default 24 hours
php artisan maintenance:cleanup-temp --older-than=7d

# Keep files for 2 hours only
php artisan maintenance:cleanup-temp --older-than=2h

# Keep files for 168 hours (7 days) - numeric format
php artisan maintenance:cleanup-temp --older-than=168
```

**Result:** Adjusts the retention policy for this execution only.

**Supported formats:**
- `24h` - Hours (e.g., 1h, 12h, 48h)
- `7d` - Days (e.g., 1d, 7d, 30d)
- `168` - Raw hours (numeric)

---

### 5. Aggressive Cleanup (Small Retention)

```bash
php artisan maintenance:cleanup-temp --older-than=1h --dry-run
```

**Result:** Shows all files older than 1 hour. Useful for emergency cleanup.

**⚠️ Warning:** Use with caution. Always run `--dry-run` first.

---

### 6. Large Batch Processing

```bash
php artisan maintenance:cleanup-temp --batch=1000
```

**Result:** Processes 1000 files per batch instead of default 500.

**Use case:** When you have many small files and want faster processing.

---

### 7. Extended Runtime

```bash
php artisan maintenance:cleanup-temp --max-runtime=300
```

**Result:** Allows the command to run for up to 5 minutes.

**Use case:** When processing very large directories with many files.

---

## 🔧 Configuration

### File Location

`config/maintenance.php`

### Configuration Structure

```php
'temp_cleanup' => [
    'enabled' => true,
    
    // Directories to scan for cleanup
    'paths' => [
        storage_path('app/temp'),
        storage_path('framework/testing'),
    ],
    
    // File retention policy
    'retention_hours' => 24,
    
    // Safety window - skip files modified within this time
    'safety_window_minutes' => 10,
    
    // Glob patterns to exclude from cleanup
    'exclusions' => [
        '*.lock',        // Lock files
        '.gitignore',    // Git ignore files
        '.placeholder',  // Placeholder files
    ],
    
    // Files to process per batch
    'batch_size' => 500,
    
    // Maximum runtime per execution (seconds)
    'max_runtime_seconds' => 120,
    
    // Cache lock timeout (seconds)
    'lock_timeout' => 600,
    
    // Clean empty directories after file cleanup
    'clean_empty_directories' => true,
],
```

### Adding New Paths

```php
'paths' => [
    storage_path('app/temp'),
    storage_path('framework/testing'),
    storage_path('app/exports'),      // Add new path
    storage_path('logs/old'),         // Add another path
],
```

### Adding Exclusion Patterns

```php
'exclusions' => [
    '*.lock',
    '.gitignore',
    '.placeholder',
    '*.processing',    // Add new pattern
    'important_*',     // Exclude files starting with 'important_'
],
```

---

## 📅 Automatic Scheduling

### Current Schedule

The command runs **automatically every hour at minute 15**.

```php
// src/App/Console/Kernel.php
$schedule->command('maintenance:cleanup-temp')
    ->hourlyAt(15)
    ->withoutOverlapping();
```

### Verify Schedule

```bash
php artisan schedule:list
```

**Expected output:**
```
0 * * * *  php artisan maintenance:cleanup-node-data ..... Next Due: 58 minutes
15 * * * * php artisan maintenance:cleanup-temp ........... Next Due: 13 minutes
```

### Modify Schedule

#### Run more frequently (every 30 minutes)

```php
$schedule->command('maintenance:cleanup-temp')
    ->everyThirtyMinutes()
    ->withoutOverlapping();
```

#### Run daily at 2 AM

```php
$schedule->command('maintenance:cleanup-temp')
    ->dailyAt('02:00')
    ->withoutOverlapping();
```

#### Run with custom retention

```php
$schedule->command('maintenance:cleanup-temp --older-than=12h')
    ->hourly()
    ->withoutOverlapping();
```

---

## 📊 Monitoring and Telemetry

### Log Location

Logs are written to the configured logging channel (default: `daily`).

**File:** `storage/logs/laravel-YYYY-MM-DD.log`

### Log Structure

```json
{
  "message": "Temporary files cleanup completed",
  "context": {
    "category": "storage_cleanup",
    "paths_processed": 2,
    "files_scanned": 143,
    "files_deleted": 143,
    "bytes_reclaimed": 47781376,
    "empty_dirs_removed": 3,
    "duration_ms": 1012,
    "dry_run": false,
    "errors": 0,
    "batch_size": 500
  }
}
```

### Query Logs

```bash
# View today's cleanup logs
tail -f storage/logs/laravel-$(date +%Y-%m-%d).log | grep "storage_cleanup"

# Count cleanups in last 7 days
grep "storage_cleanup" storage/logs/laravel-*.log | wc -l

# Check for errors
grep "storage_cleanup" storage/logs/laravel-*.log | grep "errors"
```

### Metrics to Monitor

1. **Files Deleted per Run**
   - Normal: 50-200 files
   - High: >500 files (investigate why so many temp files)
   - Low: <10 files (good, system is clean)

2. **Space Reclaimed**
   - Monitor trend over time
   - Alert if consistently >1GB (potential issue)

3. **Errors**
   - Should be 0 most of the time
   - Investigate any non-zero errors

4. **Duration**
   - Normal: 1-5 seconds
   - High: >30 seconds (may need optimization)

---

## 🛡️ Safety Features

### 1. **Symlink Protection**
- Symlinks are skipped by default
- Prevents accidental deletion of linked files
- Configurable via `LocalStorageService`

### 2. **Lock File Detection**
- Files with `.lock` suffix are excluded
- Files with sibling `.lock` files are skipped
- Example: `data.csv` skipped if `data.csv.lock` exists

### 3. **Safety Window**
- Recent files (< 10 minutes old) are never deleted
- Protects files being actively written
- Prevents race conditions

### 4. **Batch Processing**
- Deletes in small batches (default 500)
- Reduces filesystem stress
- Allows for graceful interruption

### 5. **Modification Time Verification**
- Verifies file hasn't been modified since scan
- Skips if modification time changed
- Prevents deletion of files updated during scan

### 6. **Cache Lock**
- Only one instance can run at a time
- Prevents concurrent executions
- Auto-releases after completion or timeout

### 7. **Max Runtime**
- Stops gracefully after configured time
- Prevents indefinite execution
- Resumes on next scheduled run

---

## 🚨 Troubleshooting

### Problem: Files Not Being Deleted

**Symptoms:**
- `--dry-run` shows candidates but actual run deletes nothing
- Log shows "0 files deleted"

**Possible Causes:**

1. **Files within retention period**
   ```bash
   # Check file ages
   ls -lh --time-style=long-iso storage/app/temp/
   
   # Verify they're older than retention
   php artisan maintenance:cleanup-temp --older-than=1h --dry-run
   ```

2. **Files within safety window**
   ```bash
   # Check recently modified files
   find storage/app/temp/ -mmin -10
   ```

3. **Lock files present**
   ```bash
   # Check for lock files
   find storage/app/temp/ -name "*.lock"
   ```

4. **Permission issues**
   ```bash
   # Check file permissions
   ls -la storage/app/temp/
   
   # Fix permissions
   chmod 755 storage/app/temp/
   chmod 644 storage/app/temp/*
   ```

---

### Problem: Command Not Running on Schedule

**Symptoms:**
- No log entries
- Files accumulating

**Solutions:**

1. **Verify cron is running**
   ```bash
   # Check if scheduler is running
   ps aux | grep schedule:run
   
   # Check cron logs
   grep CRON /var/log/syslog
   ```

2. **Verify schedule is configured**
   ```bash
   php artisan schedule:list | grep cleanup-temp
   ```

3. **Run manually to test**
   ```bash
   php artisan maintenance:cleanup-temp -v
   ```

4. **Check lock isn't stuck**
   ```bash
   php artisan tinker
   >>> Cache::forget('maintenance:cleanup-temp');
   ```

---

### Problem: Cleanup Too Aggressive

**Symptoms:**
- Important files being deleted
- Users reporting missing files

**Solutions:**

1. **Increase retention period**
   ```php
   // config/maintenance.php
   'retention_hours' => 48,  // Instead of 24
   ```

2. **Add exclusion patterns**
   ```php
   'exclusions' => [
       '*.lock',
       'important_*',
       'export_final_*',
   ],
   ```

3. **Move important files out of temp directories**
   ```bash
   # Important files should not be in temp/
   mv storage/app/temp/important.csv storage/app/permanent/
   ```

---

### Problem: Storage Still Full

**Symptoms:**
- Cleanup runs successfully
- Storage usage still high

**Investigation:**

1. **Check what's using space**
   ```bash
   du -sh storage/*
   du -sh storage/app/*
   du -sh storage/logs/*
   ```

2. **Check for large files not in cleanup paths**
   ```bash
   find storage/ -type f -size +100M
   ```

3. **Check database files**
   ```bash
   du -sh storage/*.sqlite
   du -sh database/*.sqlite
   ```

**Solutions:**

1. **Add more paths to cleanup**
   ```php
   'paths' => [
       storage_path('app/temp'),
       storage_path('framework/testing'),
       storage_path('logs/old'),        // Add logs
       storage_path('app/exports/old'), // Add old exports
   ],
   ```

2. **Reduce retention**
   ```php
   'retention_hours' => 12,  // More aggressive
   ```

3. **Manual cleanup of other areas**
   ```bash
   # Clean old logs
   find storage/logs -name "*.log" -mtime +7 -delete
   
   # Clean old database files
   find storage -name "*.sqlite" -mtime +1 -delete
   ```

---

### Problem: Command Times Out

**Symptoms:**
- "Max runtime exceeded" message
- Cleanup incomplete

**Solutions:**

1. **Increase max runtime**
   ```php
   'max_runtime_seconds' => 300,  // 5 minutes
   ```

2. **Increase batch size**
   ```php
   'batch_size' => 1000,  // Process more files per batch
   ```

3. **Run more frequently**
   ```php
   $schedule->command('maintenance:cleanup-temp')
       ->everyThirtyMinutes()  // More frequent, smaller batches
   ```

4. **Optimize filesystem**
   ```bash
   # Check filesystem performance
   time ls -la storage/app/temp/ > /dev/null
   
   # Consider moving to faster storage if slow
   ```

---

## 📈 Best Practices

### 1. **Always Test with --dry-run First**

```bash
# Before production cleanup
php artisan maintenance:cleanup-temp --older-than=1h --dry-run

# If output looks good, run for real
php artisan maintenance:cleanup-temp --older-than=1h
```

### 2. **Monitor Logs Regularly**

```bash
# Set up daily log check
grep "storage_cleanup" storage/logs/laravel-$(date +%Y-%m-%d).log
```

### 3. **Adjust Retention Based on Usage**

- **Development:** 12-24 hours
- **Staging:** 24-48 hours  
- **Production:** 48-72 hours

### 4. **Document Custom Configurations**

```php
// config/maintenance.php
'retention_hours' => 48,  // Increased due to long-running exports
'exclusions' => [
    '*.lock',
    'export_final_*',  // Keep final exports longer
],
```

### 5. **Use Separate Paths for Different File Types**

```
storage/app/temp/
├── exports/     # CSV/Excel exports
├── uploads/     # Temporary uploads
├── processing/  # Files being processed
└── cache/       # Cached data
```

### 6. **Set Up Alerts**

```bash
# Alert if cleanup deletes >1000 files
if [ $(grep "files_deleted" storage/logs/laravel-*.log | tail -1 | grep -oP '\d+') -gt 1000 ]; then
    echo "Warning: Unusually high file cleanup" | mail -s "Cleanup Alert" admin@example.com
fi
```

---

## 🔗 Related Commands

### Node Data Cleanup

```bash
php artisan maintenance:cleanup-node-data
```

Cleans up old `node_data` table records (database cleanup).

### Log Rotation

```bash
# Laravel log rotation (if configured)
php artisan log:clear
```

### Cache Clearing

```bash
php artisan cache:clear
php artisan config:clear
php artisan route:clear
```

---

## 📚 Additional Resources

- **Service Documentation:** `docs/services/LocalStorageService.md`
- **Test Artifacts:** `docs/maintenance/test-artifacts-organization.md`
- **Configuration:** `config/maintenance.php`
- **Source Code:** `src/App/Console/Commands/MaintenanceTempCleanUp.php`

---

## 💡 Tips

1. **Use `--dry-run` liberally** - It's free and shows you exactly what will happen
2. **Monitor space regularly** - `du -sh storage/app/temp/`
3. **Adjust retention based on needs** - Not all files need 24 hours
4. **Keep exclusions updated** - Add patterns for important files
5. **Check logs after changes** - Verify cleanup is working as expected
6. **Test in staging first** - Especially when changing retention policies

---

## ✅ Quick Reference

```bash
# Most common commands
php artisan maintenance:cleanup-temp                    # Normal cleanup
php artisan maintenance:cleanup-temp --dry-run          # Preview
php artisan maintenance:cleanup-temp --older-than=12h   # Custom retention
php artisan maintenance:cleanup-temp --path=storage/app/temp  # Specific path

# Check status
php artisan schedule:list | grep cleanup-temp           # Verify schedule
du -sh storage/app/temp                                 # Check space
find storage/app/temp -type f | wc -l                   # Count files

# Emergency cleanup
php artisan maintenance:cleanup-temp --older-than=1h --dry-run  # Check
php artisan maintenance:cleanup-temp --older-than=1h            # Execute
```

---

## 📞 Support

If you encounter issues not covered in this guide:

1. Check logs: `storage/logs/laravel-*.log`
2. Run with `-v` flag for verbose output
3. Review configuration: `config/maintenance.php`
4. Check file permissions and ownership
5. Verify filesystem has adequate free space

---

**Last Updated:** October 2, 2025  
**Version:** 1.0.0  
**Maintainer:** Development Team