Skip to content

Filter Functionality

Example Code: examples/knowledge/features/metadata-filter and examples/knowledge/features/agentic-filter

The Knowledge system provides powerful filter functionality, allowing precise search based on document metadata. This includes static filters (manually preset) and intelligent filters (automatically generated by LLM based on document metadata).

Configuring Metadata Sources

For filter functionality to work properly, metadata needs to be added when creating document sources:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
import (
    "trpc.group/trpc-go/trpc-agent-go/knowledge/source"
    filesource "trpc.group/trpc-go/trpc-agent-go/knowledge/source/file"
    dirsource "trpc.group/trpc-go/trpc-agent-go/knowledge/source/dir"
    urlsource "trpc.group/trpc-go/trpc-agent-go/knowledge/source/url"
)

sources := []source.Source{
    // File source with metadata
    filesource.New(
        []string{"./docs/api.md"},
        filesource.WithName("API Documentation"),
        filesource.WithMetadataValue("category", "documentation"),
        filesource.WithMetadataValue("topic", "api"),
        filesource.WithMetadataValue("service_type", "gateway"),
        filesource.WithMetadataValue("protocol", "trpc-go"),
        filesource.WithMetadataValue("version", "v1.0"),
    ),

    // Directory source with metadata
    dirsource.New(
        []string{"./tutorials"},
        dirsource.WithName("Tutorials"),
        dirsource.WithMetadataValue("category", "tutorial"),
        ...
    ),

    // URL source with metadata
    urlsource.New(
        []string{"https://example.com/wiki/rpc"},
        urlsource.WithName("RPC Wiki"),
        urlsource.WithMetadataValue("category", "encyclopedia"),
        ...
    ),
}

Tip: For more document source configuration details, see Document Source Configuration.

Filter Field Naming Convention

When using filters, metadata fields require the metadata. prefix:

  • The metadata. prefix distinguishes metadata fields from system fields (like id, name, content, etc.)
  • Whether setting conditions for basic filters or generating conditions for intelligent filters, metadata field names must include this prefix.
  • Basic Filters: e.g., searchfilter.Equal("metadata.category", "docs")
  • Intelligent Filters: The JSON field names generated by LLM must also include the prefix, e.g., {"metadata.topic": "api"} (system handles this automatically).
  • Exception: Custom table fields added via WithDocBuilder (like status, priority, etc.) use the field name directly without prefix.

Filter Hierarchy

The Knowledge system supports multiple filter levels, all filters are implemented using FilterCondition and combined using AND logic. The system doesn't distinguish priority; all levels of filters are merged equally.

Filter Hierarchy:

  1. Tool Level Filters / Agent Level Filters:

    • Manually Created Tools (injected via llmagent.WithTools(customSearchTool)): Set via knowledgetool.WithConditionedFilter().
    • Automatically Created Tools (injected via llmagent.WithKnowledge): Set via llmagent.WithKnowledgeConditionedFilter().
    • Note: Both are essentially the same, acting as static filters on the Tool instance.

  2. Runner Level Filters:

    • Pass conditioned filters via agent.WithKnowledgeConditionedFilter() when calling runner.Run().

  3. LLM Intelligent Filters:

    • Filter conditions dynamically generated by LLM based on user queries.

Important Note:

  • All filters are combined using AND logic, meaning all filter conditions from all levels must be satisfied simultaneously.

Basic Filters

Basic filters support two ways of setting: Tool/Agent-level fixed filters and Runner-level runtime filters.

Tool/Agent Level Filters

Way 1: Agent-Level Filters when Automatically Creating Tools

When creating an Agent and automatically injecting the Knowledge Tool via WithKnowledge, you can preset fixed search filter conditions using WithKnowledgeConditionedFilter:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
import (
    "trpc.group/trpc-go/trpc-agent-go/agent/llmagent"
    "trpc.group/trpc-go/trpc-agent-go/knowledge/searchfilter"
)

// Create Agent with fixed filters
llmAgent := llmagent.New(
    "knowledge-assistant",
    llmagent.WithModel(modelInstance),
    llmagent.WithKnowledge(kb),
    llmagent.WithKnowledgeConditionedFilter(
        searchfilter.And(
            searchfilter.Equal("metadata.category", "documentation"),
            searchfilter.Equal("metadata.topic", "api"),
        ),
    ),
)

Way 2: Tool-Level Filters when Manually Creating Tools

When manually creating a Knowledge Search Tool, you can inject filters using WithConditionedFilter:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
import (
    "trpc.group/trpc-go/trpc-agent-go/agent/llmagent"
    "trpc.group/trpc-go/trpc-agent-go/knowledge/searchfilter"
    knowledgetool "trpc.group/trpc-go/trpc-agent-go/knowledge/tool"
)

// Manually create Tool and set filters
searchTool := knowledgetool.NewKnowledgeSearchTool(
    kb,
    knowledgetool.WithConditionedFilter(
        searchfilter.Equal("metadata.language", "en"),
    ),
)

// Inject Tool into Agent
llmAgent := llmagent.New(
    "knowledge-assistant",
    llmagent.WithModel(modelInstance),
    llmagent.WithTools(searchTool),
)

Runner-Level Filters

Dynamically pass filters when calling runner.Run(), suitable for scenarios requiring filtering based on different request contexts:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
import (
    "trpc.group/trpc-go/trpc-agent-go/agent"
    "trpc.group/trpc-go/trpc-agent-go/knowledge/searchfilter"
    "trpc.group/trpc-go/trpc-agent-go/runner"
)

// Create Runner
appRunner := runner.NewRunner("knowledge-chat", llmAgent)

// Pass filters at runtime
eventCh, err := appRunner.Run(
    ctx,
    userID,
    sessionID,
    message,
    agent.WithKnowledgeConditionedFilter(
        searchfilter.And(
            searchfilter.Equal("metadata.category", "tutorial"),
            searchfilter.Equal("metadata.difficulty", "beginner"),
            searchfilter.Equal("metadata.language", "zh"),
        ),
    ),
)

Filter Merge Rules

Tool/Agent-level filters and Runner-level filters are combined using AND logic:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
import (
    "trpc.group/trpc-go/trpc-agent-go/agent"
    "trpc.group/trpc-go/trpc-agent-go/agent/llmagent"
    "trpc.group/trpc-go/trpc-agent-go/knowledge/searchfilter"
    "trpc.group/trpc-go/trpc-agent-go/runner"
)

// Tool/Agent-level filter (metadata.category = "documentation")
llmAgent := llmagent.New(
    "assistant",
    llmagent.WithKnowledge(kb),
    llmagent.WithKnowledgeConditionedFilter(
        searchfilter.And(
            searchfilter.Equal("metadata.category", "documentation"),
            searchfilter.Equal("metadata.source_type", "web"),
        ),
    ),
)

// Create Runner
appRunner := runner.NewRunner("knowledge-chat", llmAgent)

// Runner-level filter (metadata.topic = "api")
eventCh, err := appRunner.Run(
    ctx, userID, sessionID, message,
    agent.WithKnowledgeConditionedFilter(
        searchfilter.Equal("metadata.topic", "api"),
    ),
)

// Final effective filter (AND combination):
// metadata.category = "documentation" AND
// metadata.source_type = "web" AND
// metadata.topic = "api"

Intelligent Filters (Agentic Filter)

Intelligent filters are an advanced feature of the Knowledge system, allowing LLM Agents to dynamically select appropriate filter conditions based on user queries.

Enabling Intelligent Filters

When enabling intelligent filters, you need to provide available metadata field information via WithKnowledgeAgenticFilterInfo. This information serves as part of the prompt to guide the LLM in generating correct filter conditions.

Three configuration methods are supported:

Way 1: Automatic Extraction (Recommended)

Automatically extract metadata information from configured document sources:

  1. Extract Fields and Values: Use source.GetAllMetadata(sources), LLM will select from the extracted enum values (suitable for finite enum values).
  2. Extract Fields Only: Use source.GetAllMetadataWithoutValues(sources), LLM will infer values freely based on user queries (suitable for open-domain values).
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
import (
    "trpc.group/trpc-go/trpc-agent-go/agent/llmagent"
    "trpc.group/trpc-go/trpc-agent-go/knowledge/source"
)

// 1. Extract all metadata info (including field names and all occurred values)
// e.g. {"metadata.category": ["doc", "tutorial"], "metadata.topic": ["api", "rpc"]}
sourcesMetadata := source.GetAllMetadata(sources)

// 2. Or extract field names only (unlimited values, inferred by LLM)
// e.g. {"metadata.category": [], "metadata.topic": []}
// sourcesMetadata := source.GetAllMetadataWithoutValues(sources)

llmAgent := llmagent.New(
    // ...
    llmagent.WithEnableKnowledgeAgenticFilter(true),
    llmagent.WithKnowledgeAgenticFilterInfo(sourcesMetadata),
)

Way 2: Manual Configuration (Specify Fields and Values)

Manually specify allowed filter fields and their enum values, suitable for scenarios with limited enum values (e.g., status, category):

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
// Manually specify fields and optional values
customMetadata := map[string][]string{
    "category": {"documentation", "tutorial", "blog"},
    "language": {"en", "zh"},
}

llmAgent := llmagent.New(
    // ...
    llmagent.WithEnableKnowledgeAgenticFilter(true),
    llmagent.WithKnowledgeAgenticFilterInfo(customMetadata),
)

Way 3: Manual Configuration (Specify Fields, Infer Values by LLM)

Only specify allowed filter fields, leave value lists empty (nil or empty slice), letting LLM infer values freely (suitable for fields with too many enum values like ID, name):

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
// Only specify fields, no limit on value range
customMetadata := map[string][]string{
    "author_id": nil,       // LLM automatically extracts author_id
    "publish_year": nil,    // LLM automatically extracts year
    "category": {"news"},   // Mixed use: category limited to "news"
}

llmAgent := llmagent.New(
    // ...
    llmagent.WithEnableKnowledgeAgenticFilter(true),
    llmagent.WithKnowledgeAgenticFilterInfo(customMetadata),
)

Filter Combination Example

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
import (
    "trpc.group/trpc-go/trpc-agent-go/agent"
    "trpc.group/trpc-go/trpc-agent-go/knowledge/searchfilter"
    "trpc.group/trpc-go/trpc-agent-go/runner"
)


// 2. Runner-level filter
appRunner := runner.NewRunner("knowledge-chat", llmAgent)
eventCh, err := appRunner.Run(
    ctx, userID, sessionID, message,
    agent.WithKnowledgeConditionedFilter(
        searchfilter.And(
            searchfilter.Equal("metadata.language", "zh"),
            searchfilter.Equal("metadata.version", "v1.0"),
        ),
    ),
)

// 3. LLM intelligent filter (dynamically generated by LLM)
// Example: User asks "Find API related documents", LLM might generate {"field": "metadata.topic", "value": "api"}

// Final effective filter conditions (all conditions combined with AND):
// metadata.source_type = "web" AND
// metadata.category = "documentation" AND
// metadata.protocol = "trpc-go" AND
// metadata.language = "zh" AND
// metadata.version = "v1.0" AND
// metadata.topic = "api"
//
// Meaning: All conditions from all levels must be satisfied simultaneously

Complex Condition Filters

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
import (
    "trpc.group/trpc-go/trpc-agent-go/agent/llmagent"
    "trpc.group/trpc-go/trpc-agent-go/knowledge/searchfilter"
    knowledgetool "trpc.group/trpc-go/trpc-agent-go/knowledge/tool"
)

// Manually create Tool with conditioned filter
searchTool := knowledgetool.NewKnowledgeSearchTool(
    kb,
    knowledgetool.WithConditionedFilter(
        searchfilter.And(
            searchfilter.Equal("metadata.source_type", "web"),
            searchfilter.Or(
                searchfilter.Equal("metadata.topic", "programming"),
                searchfilter.Equal("metadata.topic", "api"),
            ),
        ),
    ),
)

llmAgent := llmagent.New(
    "knowledge-assistant",
    llmagent.WithModel(modelInstance),
    llmagent.WithTools(searchTool),  // Manually pass Tool
)

// Final filter condition:
// metadata.source_type = "web" AND (metadata.topic = "programming" OR metadata.topic = "api")
// Meaning: Must be web source, and topic is either programming or API

Common Filter Helper Functions

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
import "trpc.group/trpc-go/trpc-agent-go/knowledge/searchfilter"

// Comparison operators (Note: metadata fields need metadata. prefix)
searchfilter.Equal("metadata.topic", value)                  // metadata.topic = value
searchfilter.NotEqual("metadata.category", value)            // metadata.category != value
searchfilter.GreaterThan("metadata.version", value)          // metadata.version > value
searchfilter.GreaterThanOrEqual("metadata.version", value)   // metadata.version >= value
searchfilter.LessThan("metadata.version", value)             // metadata.version < value
searchfilter.LessThanOrEqual("metadata.version", value)      // metadata.version <= value
searchfilter.In("metadata.category", values...)              // metadata.category IN (...)
searchfilter.NotIn("metadata.topic", values...)              // metadata.topic NOT IN (...)
searchfilter.Like("metadata.protocol", pattern)              // metadata.protocol LIKE pattern
searchfilter.Between("metadata.version", min, max)           // metadata.version BETWEEN min AND max

// Custom table fields (extra columns added via WithDocBuilder) don't need prefix
searchfilter.NotEqual("status", "deleted")                   // status != "deleted"
searchfilter.GreaterThanOrEqual("priority", 3)               // priority >= 3

// Logical operators
searchfilter.And(conditions...)               // AND combination
searchfilter.Or(conditions...)                // OR combination

// Nested example: (metadata.category = 'documentation') AND (metadata.topic = 'api' OR metadata.topic = 'rpc')
searchfilter.And(
    searchfilter.Equal("metadata.category", "documentation"),
    searchfilter.Or(
        searchfilter.Equal("metadata.topic", "api"),
        searchfilter.Equal("metadata.topic", "rpc"),
    ),
)

Multiple Document Return

Knowledge Search Tool supports returning multiple relevant documents, configurable via WithMaxResults(n) option:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
import knowledgetool "trpc.group/trpc-go/trpc-agent-go/knowledge/tool"

// Create search tool, limit to at most 5 documents
searchTool := knowledgetool.NewKnowledgeSearchTool(
    kb,
    knowledgetool.WithMaxResults(5),
)

// Or use intelligent filter search tool
agenticSearchTool := knowledgetool.NewAgenticFilterSearchTool(
    kb,
    sourcesMetadata,
    knowledgetool.WithMaxResults(10),
)

Each returned document contains text content, metadata, and relevance score, sorted by score in descending order.