kb-retriever

Local Knowledge Base Retrieval Skill (kb-retriever)

Knowledge Base Directory Description

• The knowledge base is stored in a root directory, containing multiple file types (e.g., .md/.txt, .pdf, .xlsx), usually organized into multi-level subdirectories by type or business purpose.
• Uses hierarchical directory index files:
  • The root directory has a data_structure.md describing the main "domain directories" and their purposes.
  • Each domain directory can have its own data_structure.md describing subdirectories/files and their purposes.
  • Deeper subdirectories can also have data_structure.md, forming a multi-level index tree.
• Knowledge base root directory convention:
  • Default: the knowledge base is located at knowledge/ under the current project root.
  • If the user explicitly specifies another path (e.g., "my knowledge base is in /data/kb" or "use ./docs as the knowledge base"), use that path as the root.
  • When the default path knowledge/ does not exist or is inaccessible, ask the user for the actual knowledge base root directory instead of guessing.
• Individual business files may be large:
  • Do not use Read to load the entire file directly.
  • For PDF and Excel, use the corresponding skill to process them structurally, then combine with grep/partial reads for fine-grained retrieval.

Locating the knowledge Root Directory

• Prioritize user input: if the user provides a path (e.g., ./docs, ./knowledge-personal), use it directly.
• Default root: otherwise, the default root is knowledge/ under the current project.
  • Use shell to explicitly check if the directory exists: prefer test -d knowledge, or fall back to ls -d knowledge.
  • Note: Do NOT use Glob "knowledge" in . or similar patterns to check directory existence. Glob only returns file paths, not the directory itself; an empty result cannot distinguish between "directory does not exist" and "directory exists but is empty".
• Only use Glob to search within the root directory after confirming its existence via test -d or similar, and set the directory as path, e.g.:
  • Index files: pattern="**/data_structure.md", path="knowledge"
  • All Markdown: pattern="**/*.md", path="knowledge"
• If the default knowledge/ does not exist (test -d fails): do not guess other directories; explicitly tell the user the default root was not found and ask them to specify the actual path.

Key Principle: Learn First, Then Process

Mandatory checklist when encountering PDF or Excel files:

• [ ] ✅ Read the corresponding references document to learn processing methods
• [ ] ✅ Understand the recommended tools and commands
• [ ] ✅ Complete file processing (extraction/conversion)
• [ ] ⏭️ Now you can start retrieval

Prohibited actions:

• ❌ Attempting to process a PDF without first reading pdf_reading.md
• ❌ Attempting to process an Excel without first reading excel_reading.md
• ❌ Skipping file processing and directly retrieving from raw PDF/Excel

Overall Workflow

1. Understand user needs

   • Read the user question and extract:
     • Topic/domain keywords (e.g., "sales report", "system architecture", "API documentation")
     • Time or scope constraints (e.g., "2023 Q1", "latest version")
     • Required output type (explanation, summary, specific field values, etc.)
   • Determine the knowledge base root directory:
     • First check if the user specified a path in the question.
     • Otherwise, use the default root knowledge/.
     • If the default root does not exist or the directory structure is abnormal, ask the user for confirmation instead of assuming.

2. Hierarchically view directory index data_structure.md

   • Use a "current working directory" concept:
     • Start from the user-specified knowledge base root; if not specified, use the current directory.
   • If data_structure.md exists in the current working directory:
     • Use Read to read the first few lines (e.g., limit=300), continuing in segments if necessary.
     • Goal:
       • Understand which subdirectories and files exist under the current directory.
       • Understand the purpose of each subdirectory/file.
     • Based on the user question, select the most relevant subdirectories or files as candidates.
   • For candidate subdirectories:
     • Recursively enter that subdirectory as the new "current working directory" and look for its data_structure.md, repeating the process.
     • During recursion, avoid diving into all branches at once; prioritize the path most relevant to the question.
   • For candidate business files (md/text, PDF, Excel, etc.):
     • After exploring the necessary directory levels, collect these files as the final retrieval target list.
   • When prioritizing:
     • Prefer domain directories and files whose purpose description closely matches the question topic.
     • Then consider time/version constraints (if reflected in the index).
     • General documentation (e.g., README.md, overall design docs) should have lower priority.

3. Learn file processing methods (mandatory for PDF/Excel)

   • Before processing PDF files:
     • Must first read references/pdf_reading.md (note: this directory is under Skills, not Knowledge) to learn extraction methods.
     • Focus on: pdftotext command, pdfplumber usage, table extraction methods.
   • Before processing Excel files:
     • Must first read references/excel_reading.md to learn reading methods.
     • Must first read references/excel_analysis.md to learn analysis methods.
     • Focus on: pandas reading, column filtering, data filtering.
   • Purpose: Ensure correct tools and methods are used, avoiding blind retrieval.

4. Execute processing and retrieval by file type

   • Use the methods just learned to process files (extract, convert, structure).
   • For each candidate file type, follow the strategies below for "Markdown/Text", "PDF", and "Excel".
   • General principles:
     • Start with the most relevant and precise files.
     • Within each file, progressively retrieve locally, avoiding loading the entire content at once.
     • If the current file does not yield satisfactory information, switch to the next candidate.

5. Iterative retrieval

   • All file types use a unified "multi-round iterative retrieval mechanism" (see common retrieval principles above).

6. Answer organization and source tracing

   • Aggregate context from multiple retrieval rounds to answer the user's question comprehensively.
   • Try to:
     • Provide clear, direct answers.
     • Indicate the files used (including approximate location, such as section or line/page number if necessary).
   • If the answer is based on inference or incomplete information:
     • Clearly mark assumptions and uncertainties.
     • Suggest that the user can provide more specific file ranges or keywords.

Common Retrieval Principles

Keyword Selection Strategy

• Extract 3-8 keywords from the user question (including possible English abbreviations, synonyms, hypernyms/hyponyms).
• Combine phrases (e.g., "sales report", "API timeout").
• Include business terms, technical jargon, common abbreviations (e.g., "UV", "PV", "GMV") when necessary.

Basic grep Retrieval Principles

• Always specify as precise include and path as possible to avoid searching the entire directory.
• For pattern, first try the core nouns and terms from the question, then try synonyms.
• For each match, only read the local area around the match (a few lines above and below).
• Save "filename + location info + text snippet".

Multi-round Iterative Retrieval Mechanism (max 5 rounds)

All file types use a unified iterative strategy:

1. Iteration control
   • Maintain a "retry count" counter, max 5 rounds.
   • Increment after each retrieval.
2. Each iteration flow
   1. Generate/update retrieval keywords based on the question (including synonyms, expanded terms).
   2. Select files or file parts not yet fully retrieved.
   3. Execute retrieval (grep/partial read/specialized skill call).
   4. Analyze the obtained context snippets.
   5. Determine if enough to answer the question.
3. Termination conditions
   • Sufficient context found to support an answer; or
   • Reached 5 attempts without finding suitable information.
4. Handling insufficient information
   • Clearly inform the user that information is missing or may not be in the current knowledge base.
   • Provide the closest information found and explain uncertainties.
   • Suggest how the user can narrow the scope (more specific filenames, keywords, time range, etc.).

Notes

• Do NOT directly call Glob "knowledge" in . or any call that uses Glob to check directory existence. Directory existence should be checked via shell commands (e.g., test -d).
• When using this skill to query the knowledge base, do NOT use other tools like web search to obtain knowledge.

Specific Strategies by File Type

1. Markdown / Text Files (.md, .txt, .log, etc.)

1. Candidate file selection

   • Determine relevance based on data_structure.md, filename, and path.
   • Prioritize title and index files (e.g., summary documents, design overviews).

2. grep localization and partial reading

   • Use Grep tool on specified candidate files, with include limited to specific extensions (e.g., "*.md").
   • For files with matches, use Read to read only the local area around the match:
     • Control reading via line offset and limit (e.g., read a few dozen lines before and after the match line).
     • Avoid reading the entire file.

3. Special handling

   • If content is only a table of contents/titles, continue locating deeper content based on links or section names.
   • Apply the "multi-round iterative retrieval mechanism" (see common retrieval principles above).

2. PDF File Retrieval Strategy

Workflow:

1. First: Read processing method guide

   • Before processing any PDF, must first read references/pdf_reading.md (note: this directory is under Skills, not Knowledge).
   • Focus on: pdftotext command, pdfplumber usage, table extraction methods, quick decision table.

2. Select candidate PDFs

   • Based on descriptions in data_structure.md, select 1-3 most relevant files.
   • If the user specifies a specific PDF file, prioritize that file.

3. Apply learned methods to extract text

   • Use tools recommended in pdf_reading.md (prefer pdftotext or pdfplumber).
   • Important: Use pdftotext input.pdf output.txt to extract text to a file, not to stdout (to avoid consuming many tokens).
   • For table extraction, use pdfplumber's table extraction feature.

4. Perform retrieval on extracted results

   • Use grep to search keywords in the extracted text.
   • For each match, extract context around the match (tens of lines above/below or adjacent pages).
   • Save "filename + page/approximate location + text snippet".
   • Apply the "multi-round iterative retrieval mechanism" (see common retrieval principles above).

3. Excel File Retrieval Strategy

Workflow:

1. First: Read processing method guides

   • Before processing any Excel, must first read:
     • references/excel_reading.md - Learn how to read worksheets (note: this directory is under Skills, not Knowledge).
     • references/excel_analysis.md - Learn how to analyze data (note: this directory is under Skills, not Knowledge).
   • Focus on: pandas reading methods, column filtering, data filtering, aggregation operations.

2. Select candidate Excel files

   • Based on data_structure.md and file/worksheet naming, select the most relevant tables.
   • Prefer workbooks/worksheets containing keywords like "report", "statistics", "log", "configuration", "mapping".
   • If the user specifies a specific Excel file, prioritize that file.

3. Apply learned methods to explore structure

   • Use pandas to read the first 10-50 rows (use nrows parameter to limit).
   • Focus on: column names/field names, data types (numeric, date, text), key fields.
   • Compare column names with the user question to identify potential key fields (e.g., "revenue", "sales", "error_code").

4. Execute data retrieval and analysis

   • Use learned pandas methods for filtering and aggregation (e.g., df[df['column'] == value]).
   • Only read data near matching rows each time, avoid reading the entire table at once.
   • If the question includes a time range, add time filtering to the retrieval.
   • Apply the "multi-round iterative retrieval mechanism" (see common retrieval principles above).

Collaboration with Other Tools

PDF Processing

• Must read references/pdf_reading.md before processing PDFs to learn methods.
• Use pdfplumber/pypdf for text extraction, table extraction, metadata reading.
• Prefer pdftotext command-line tool for quick text extraction.

Excel Processing

• Must read before processing Excel:
  • references/excel_reading.md - Learn reading methods.
  • references/excel_analysis.md - Learn analysis methods.
• Use pandas for data exploration, preview, filtering, and analysis.

Tool Usage Principles

• Grep: For searching keywords in specified files to find line numbers and matching snippets; always specify as precise include and path as possible.
• Read: Only for partial file reading; always set a reasonable limit (e.g., 200-500 lines) and appropriate offset.
• For any potentially large file:
  • Do not read from start to end directly.
  • Always narrow down scope via index, table of contents, keywords, etc., before reading.

Answer Style and Error Handling

• Answer style
  • Answer in the user's language (Chinese/English) as much as possible.
  • Give the conclusion first, then a brief rationale.
  • If needed, list referenced files and approximate locations, e.g.:
    • Source: design/api_gateway.md around line 100
    • Source: reports/2023_Q1_sales.xlsx Summary worksheet
• When information is missing or uncertain
  • Clearly state that no exact match was found in the current knowledge base, or only a partial answer is possible.
  • Do not fabricate facts.
  • Suggest how the user can help narrow the scope:
    • Specify a more specific directory/file.
    • Provide more precise keywords or field names.
    • Specify a time/version range.