# Search Documentation

## Overview
The application implements a hybrid search system combining both text-based and semantic search capabilities using Typesense as the search engine. The search functionality is integrated across the application with a focus on tenant-aware searching.

## Components

### Configuration
- **Location**: `config/scout.php`
- **Purpose**: Configures Laravel Scout and Typesense settings
- **Key Features**:
  - Typesense connection settings
  - Model-specific collection schemas
  - Embedding configurations for semantic search
  - Search parameters per model
  - Field weight maps per model (`search-parameters.field_weights`) used to build `query_by_weights` dynamically

### Environment Setup
Add the following variables to your `.env` file:
```env
SCOUT_DRIVER=typesense
SCOUT_QUEUE=true

# Typesense Configuration
TYPESENSE_API_KEY=xyz  # Yes, 'xyz' for local development
```

- `SCOUT_DRIVER`: Set to 'typesense' to use Typesense as the search engine
- `SCOUT_QUEUE`: Set to true to process search operations in the background
- `TYPESENSE_API_KEY`: Your Typesense API key

### Backend Components

#### Controllers
- **SearchController** (`src/App/Http/Controllers/SearchController.php`)
  - Handles search API endpoints
  - Supports both regular and semantic search
  - Provides search statistics

#### Services
- **SearchService** (`src/App/Services/SearchService.php`)
  - Manages search index operations
  - Handles tenant-specific search context
  - Ensures per-tenant Typesense collection schema before indexing
  - Coordinates data import into search indices using a simple, predictable flow (no import-time readiness probe/backoff)

- **SemanticSearchService** (`src/App/Services/SemanticSearchService.php`)
  - Implements semantic and hybrid search functionality
  - Builds `query_by` from model base fields plus tenant custom fields
  - Builds aligned `query_by_weights` from `field_weights` and tenant registry weights
  - Excludes `embedding` from `query_by`; uses it only in `vector_query`
  - Skips models whose tenant collection does not exist
  - Hybrid fallback: if vector query fails (e.g., no embedded fields), runs text-only for that model and merges results

- **TenantService** (`src/App/Services/TenantService.php`)
  - Manages tenant context and database connections
  - Provides methods for configuring search context per tenant
  - Handles tenant database resolution and configuration

#### Jobs
- **TenantAwareMakeSearchable** (`src/App/Jobs/TenantAwareMakeSearchable.php`)
  - Extends Laravel Scout's `MakeSearchable` job
  - Captures and restores tenant context during job execution
  - Ensures Scout indexing operations use the correct tenant database
  - Overrides `restoreCollection` to set tenant context before model restoration

#### Traits
- **SearchableTenant** (`src/App/Traits/SearchableTenant.php`)
  - Extends Laravel Scout's `Searchable` trait
  - Provides tenant-aware search functionality
  - Overrides `queueMakeSearchable` to use tenant-aware jobs
  - Generates tenant-specific index names as `{tenantId}_{table}`
  - Automatically captures tenant context when dispatching Scout jobs

#### Service Providers
- **SearchServiceProvider** (`src/App/Providers/SearchServiceProvider.php`)
  - Registers search-related services and configurations
  - Handles search service registration and configuration

### Frontend Components

#### Search Widget
- **Location**: `resources/js/components/SearchWidget.js`
- **Features**:
  - Real-time search interface
  - Results categorization
  - Support for different search types (text, semantic, hybrid)
  - Configurable search parameters

#### Navigation Integration
- **Location**: `resources/views/tenant/navigation-menu.blade.php`
- **Features**:
  - Global search bar integration
  - Quick access to search functionality
  - Real-time search results display

### API Routes
- **Location**: `routes/api.php`
- **Endpoints**:
  ```php
  POST /api/search            // Regular search
  POST /api/search/semantic   // Semantic search
  GET  /api/search/stats     // Search statistics
  ```
- **Authentication**: Protected by web and auth middleware

### Search Commands
- **FlushSearchIndexes** (`src/App/Console/Commands/FlushSearchIndexes.php`)
  - Clears search indices
  - Useful for maintenance and reindexing

- **ImportSearchData** (`src/App/Console/Commands/ImportSearchData.php`)
  - Imports data into search indices
  - Handles tenant-specific data import

### Tenant Integration
The search system is tenant-aware, meaning:
- Each tenant has isolated search indices
- Search results are scoped to the current tenant
- Index names are prefixed with tenant identifiers and follow `{tenantId}_{table}`
- Scout jobs automatically maintain tenant context
- Database connections are properly configured per tenant

#### Tenant-Scoped Field Registry and Custom Fields
- Searchable custom fields per model/table are configured per tenant in `tenant_connection.search_field_configs` (columns: `model_table`, `column`, `enabled`, `weight`, `embed`).
- `TenantSearchFieldRegistry` validates enabled fields against the tenant schema (schema-first) and caches lookups per table.
- During document serialization, models using `SearchableCustomFields` include allowed custom columns, always cast as strings. Null/missing values are emitted as empty strings to satisfy Typesense schema.

## Multi-Tenant Scout Architecture

### Problem Solved
The original issue was that Laravel Scout jobs would lose tenant context when processed by the queue system, causing them to query the wrong database connection.

### Solution Overview
1. **Tenant-Aware Job System**: Created `TenantAwareMakeSearchable` job that captures tenant context during dispatch and restores it during execution
2. **Trait Override**: Modified `SearchableTenant` trait to override Scout's job dispatching at the model level
3. **Model-Level Integration**: Uses Laravel Scout's intended extension points for clean, maintainable code

### Key Components

#### TenantAwareMakeSearchable Job
```php
class TenantAwareMakeSearchable extends MakeSearchable
{
    protected $tenantId;

    public function __construct($models, $tenantId = null)
    {
        parent::__construct($models);
        $this->tenantId = $tenantId;
    }

    protected function restoreCollection($modelIdentifier)
    {
        // Set tenant context before model restoration
        if ($this->tenantId) {
            app(TenantService::class)->configureSearchForTenant($this->tenantId);
        }
        return parent::restoreCollection($modelIdentifier);
    }
}
```

#### SearchableTenant Trait
```php
trait SearchableTenant
{
    use Searchable;

    public function queueMakeSearchable($models)
    {
        $tenantId = $this->getCurrentTenantId();
        TenantAwareMakeSearchable::dispatch($models, $tenantId);
    }
}
```

## Test Coverage

### Unit Tests

#### TenantAwareMakeSearchable Tests
- **Location**: `tests/Unit/Jobs/TenantAwareMakeSearchableTest.php`
- **Coverage**:
  - Tenant context setting during job execution
  - Proper extension of Scout's MakeSearchable job
  - Graceful handling of null tenant IDs
  - Model restoration with tenant context
  - Error handling and logging

#### Model Search Configuration Tests
- **Location**: `tests/Unit/Search/ModelSearchConfigurationTest.php`
- **Coverage**:
  - Project and ProjectTask model search configurations
  - Searchable fields validation
  - Embedding fields validation
  - Schema fields configuration
  - Data type handling in searchable arrays
  - Null value handling

### Integration Tests

#### Search Service Tests
- **Location**: `tests/Feature/Search/`
- **Coverage**:
  - Tenant-specific search operations
  - Search index management
  - Data import functionality
  - Search result processing

### Test Configuration
- Uses SQLite in-memory databases for testing
- Mocks Scout engine to avoid Typesense connection issues
- Creates separate database connections for core and tenant data
- Properly sets up tenant context in test environment

## Usage

### Basic Search
```javascript
// Using the SearchWidget
const searchWidget = new SearchWidget('search-container', tenantId);
await searchWidget.initialize();
```

### Configuration
```php
// config/scout.php
'typesense' => [
    'model-settings' => [
        Project::class => [
            'collection-schema' => [
                // Collection schema configuration
            ],
            'search-parameters' => [
                // Search parameters configuration
            ]
        ]
    ]
]
```

### Adding New Searchable Models

#### 1. Model Configuration
To make a model searchable, implement the following:

1. Add Required Traits:
```php
use App\Traits\SearchableTenant;

class YourModel extends Model
{
    use SearchableTenant;
    // ... other traits
}
```

2. Define Search Methods:
```php
class YourModel extends Model
{
    /**
     * Convert model to searchable array
     */
    public function toSearchableArray()
    {
        return [
            "id" => (string) $this->id,
            "title" => $this->title,
            "created_at" => $this->created_at->timestamp,
            "updated_at" => $this->updated_at->timestamp,
            // Add model-specific fields
            "related_field" => $this->relation?->title ?? '',
        ];
    }

    /**
     * Define searchable fields
     */
    public static function getSearchableFields(): array
    {
        return [
            'title',
            'related_field',
            // Add other searchable fields
        ];
    }

    /**
     * Define fields for embedding generation
     */
    public static function getEmbeddingFields(): array
    {
        return [
            'title',
            'related_field',
            // Add fields for semantic search
        ];
    }

    /**
     * Define schema for Typesense
     */
    public static function getSchemaFields(): array
    {
        return [
            ['name' => 'id', 'type' => 'string'],
            ['name' => 'title', 'type' => 'string'],
            ['name' => 'created_at', 'type' => 'int64'],
            ['name' => 'updated_at', 'type' => 'int64'],
            // Add model-specific fields
            ['name' => 'related_field', 'type' => 'string'],
        ];
    }
}
```

#### 2. Navigation Integration
The search functionality is implemented directly in `resources/views/tenant/navigation-menu.blade.php`. When adding a new searchable model, you need to add its route to the route mapping in the navigation menu.

1. Locate the `routeMap` object in the JavaScript section of `navigation-menu.blade.php`:
```javascript
const routeMap = {
    'project': '{{ route("project.show", ":id") }}'.replace(':id', 'id='+id),
    'customer': '{{ route("customer.show", ":id") }}'.replace(':id', 'id='+id),
    'employee': '{{ route("employee.show", ":id") }}'.replace(':id', 'id='+id),
    'invoice': '{{ route("invoice.show", ":id") }}'.replace(':id', 'id='+id),
    'estimate': '{{ route("estimate.show", ":id") }}'.replace(':id', 'id='+id),
    'salesorder': '{{ route("salesorder.show", ":id") }}'.replace(':id', 'id='+id),
    'opportunity': '{{ route("opportunity.show", ":id") }}'.replace(':id', 'id='+id),
    'subscription': '{{ route("subscription.show", ":id") }}'.replace(':id', 'id='+id),
    'workorder': '{{ route("workorder.show", ":id") }}'.replace(':id', 'id='+id),
    'projecttask': '{{ route("projecttask.show", ":id") }}'.replace(':id', 'id='+id),
    'timeentry': '{{ route("timeentry.show", ":id") }}'.replace(':id', 'id='+id),
};
```

2. To add a new searchable model:
   - Add a new entry to the `routeMap`
   - Key should be the lowercase model name (e.g., 'yourmodel')
   - Value should use the Laravel route helper with the standard show route
   - Follow the pattern: `'{{ route("modelname.show", ":id") }}'.replace(':id', 'id='+id)`

3. Example adding a new model:
```javascript
const routeMap = {
    // ... existing routes ...
    'yourmodel': '{{ route("yourmodel.show", ":id") }}'.replace(':id', 'id='+id),
};
```

4. Requirements:
   - The model must have a show route defined in your routes file
   - The route name must follow the pattern `modelname.show`
   - The route must accept an `id` parameter
   - The model's record type in search results must match the routeMap key

The search functionality will automatically use this route mapping to generate links in the search results dropdown.

#### 3. Search Service Customization

If your model requires special handling of search results (like enriching with related data), extend the SemanticSearchService:

1. Add a model-specific enrichment method:
```php
protected function enrichModelWithRelatedData($models): void
{
    foreach ($models as $model) {
        $model->related_title = $model->relation?->title;
        // Add other enrichments
    }
}
```

2. Update the convertSearchResultsToModels method to use your enrichment:
```php
protected function convertSearchResultsToModels(array $searchResults, string $modelClass): Collection
{
    $models = parent::convertSearchResultsToModels($searchResults, $modelClass);

    if ($modelClass === YourModel::class) {
        $this->enrichModelWithRelatedData($models);
    }

    return $models;
}
```

Example: The Project Tasks enrichment shows how to add related data:
```php
protected function enrichProjectTasksWithProjectTitles($projectTasks): void
{
    foreach ($projectTasks as $task) {
        $task->project_title = $task->project?->title;
    }
}
```

#### 4. Testing Search Integration

After adding a new searchable model:

1. Rebuild search indices:
```bash
php artisan search:import --tenant=all
```

2. Test search functionality:
```php
// Via SemanticSearchService
$results = $searchService->semanticSearchWithinTenant(
    "search query",
    [YourModel::class]
);

// Via hybrid search
$results = $searchService->hybridSearchWithinTenant(
    "search query",
    [YourModel::class]
);
```

3. Verify results include:
   - Correct field values
   - Related data enrichments
   - Proper ranking and relevance

## Maintenance

### Reindexing Data
```bash
php artisan search:import --all         # Import all tenants
php artisan search:import --tenant=123  # Import specific tenant
```

### Clearing Indices
```bash
php artisan search:flush --all          # Clear all indices
php artisan search:flush --tenant=123   # Clear specific tenant
```

### Queue Management
```bash
php artisan queue:clear                 # Clear failed jobs
php artisan queue:work                  # Process queue jobs
```

## Security
- All search endpoints are protected by authentication
- Results are automatically scoped to the current tenant
- CSRF protection is enabled for all requests
- Search queries are validated and sanitized
- Tenant context is properly isolated

## Best Practices
1. Always use tenant-aware search operations
2. Implement proper error handling for search operations
3. Use appropriate search type based on use case:
   - Text search for exact matches
   - Semantic search for meaning-based queries
   - Hybrid search for balanced results
4. Ensure Scout queue is enabled for background processing
5. Monitor queue jobs for tenant context issues
6. Use the provided test suite to validate tenant-aware functionality

## Troubleshooting
1. Check tenant context is properly set
2. Verify search indices are properly populated
3. Ensure authentication tokens are properly set
4. Check Typesense connection and configuration
5. Monitor queue job failures for tenant context issues
6. Verify Scout queue is enabled in environment
7. Check that models use the `SearchableTenant` trait
8. Ensure proper database connections are configured per tenant
9. Hybrid error “Vector query could not find any embedded fields”:
   - Confirm the collection includes an `embedding` field, or rely on the built-in text-only fallback per model
10. If weights mismatch occurs, ensure `field_weights` maps are present; dynamic construction will normalize when misaligned