optimizer

Full name: tenets.core.distiller.optimizer

optimizer

Token optimization for context generation.

The optimizer ensures we make the best use of available tokens by intelligently selecting what to include and what to summarize.

Classes

TokenBudgetdataclass

Python

TokenBudget(total_limit: int, model: Optional[str] = None, prompt_tokens: int = 0, response_reserve: int = 4000, structure_tokens: int = 1000, git_tokens: int = 0, tenet_tokens: int = 0, _available_override: Optional[int] = None)

Manages token allocation for context building.

ATTRIBUTE	DESCRIPTION
total_limit	Total token budget available. TYPE:int
model	Optional target model name. TYPE:Optional[str]
prompt_tokens	Tokens consumed by the prompt/instructions. TYPE:int
response_reserve	Reserved tokens for model output. TYPE:int
structure_tokens	Reserved tokens for headers/formatting. TYPE:int
git_tokens	Reserved tokens for git metadata. TYPE:int
tenet_tokens	Reserved tokens for tenet injection. TYPE:int

Attributes

available_for_filespropertywritable

Python

available_for_files: int

Calculate tokens available for file content.

utilizationproperty

Python

utilization: float

Calculate budget utilization percentage.

TokenOptimizer

Python

TokenOptimizer(config: TenetsConfig)

Optimizes token usage for maximum context value.

Initialize the optimizer.

PARAMETER	DESCRIPTION
config	Tenets configuration TYPE:TenetsConfig

Source code in tenets/core/distiller/optimizer.py

Python

def __init__(self, config: TenetsConfig):
    """Initialize the optimizer.

    Args:
        config: Tenets configuration
    """
    self.config = config
    self.logger = get_logger(__name__)

Functions

create_budget

Python

create_budget(model: Optional[str], max_tokens: Optional[int], prompt_tokens: int, has_git_context: bool = False, has_tenets: bool = False) -> TokenBudget

Create a token budget for context generation.

PARAMETER	DESCRIPTION
model	Target model name. TYPE:Optional[str]
max_tokens	Optional hard cap on total tokens; overrides model default. TYPE:Optional[int]
prompt_tokens	Tokens used by the prompt/instructions. TYPE:int
has_git_context	Whether git context will be included. TYPE:boolDEFAULT:False
has_tenets	Whether tenets will be injected. TYPE:boolDEFAULT:False

RETURNS	DESCRIPTION
TokenBudget	Configured budget with reserves. TYPE:TokenBudget

Source code in tenets/core/distiller/optimizer.py

Python

def create_budget(
    self,
    model: Optional[str],
    max_tokens: Optional[int],
    prompt_tokens: int,
    has_git_context: bool = False,
    has_tenets: bool = False,
) -> TokenBudget:
    """Create a token budget for context generation.

    Args:
        model: Target model name.
        max_tokens: Optional hard cap on total tokens; overrides model default.
        prompt_tokens: Tokens used by the prompt/instructions.
        has_git_context: Whether git context will be included.
        has_tenets: Whether tenets will be injected.

    Returns:
        TokenBudget: Configured budget with reserves.
    """
    # Determine total limit
    if max_tokens:
        total_limit = max_tokens
    elif model:
        limits = get_model_limits(model)
        total_limit = limits.max_context
    else:
        total_limit = self.config.max_tokens

    # Create budget
    budget = TokenBudget(total_limit=total_limit, model=model, prompt_tokens=prompt_tokens)

    # Adjust reserves based on model
    if model and "gpt-4" in model.lower():
        budget.response_reserve = 4000
    elif model and "claude" in model.lower():
        budget.response_reserve = 4000
    else:
        budget.response_reserve = 2000

    # Reserve for git context
    if has_git_context:
        budget.git_tokens = 500  # Rough estimate

    # Reserve for tenets
    if has_tenets:
        budget.tenet_tokens = 300  # Rough estimate

    self.logger.debug(
        f"Created token budget: {budget.available_for_files:,} available for files "
        f"(total: {total_limit:,}, reserved: {total_limit - budget.available_for_files:,})"
    )

    return budget

optimize_file_selection

Python

optimize_file_selection(files: List[FileAnalysis], budget: TokenBudget, strategy: str = 'balanced') -> List[Tuple[FileAnalysis, str]]

Optimize file selection within budget.

Uses different strategies to select which files to include and whether to summarize them.

PARAMETER	DESCRIPTION
files	Ranked files to consider TYPE:List[FileAnalysis]
budget	Token budget to work within TYPE:TokenBudget
strategy	Selection strategy (greedy, balanced, diverse) TYPE:strDEFAULT:'balanced'

RETURNS	DESCRIPTION
List[Tuple[FileAnalysis, str]]	List of (file, action) tuples where action is 'full' or 'summary'

Source code in tenets/core/distiller/optimizer.py

Python

def optimize_file_selection(
    self, files: List[FileAnalysis], budget: TokenBudget, strategy: str = "balanced"
) -> List[Tuple[FileAnalysis, str]]:
    """Optimize file selection within budget.

    Uses different strategies to select which files to include
    and whether to summarize them.

    Args:
        files: Ranked files to consider
        budget: Token budget to work within
        strategy: Selection strategy (greedy, balanced, diverse)

    Returns:
        List of (file, action) tuples where action is 'full' or 'summary'
    """
    if strategy == "greedy":
        return self._greedy_selection(files, budget)
    elif strategy == "balanced":
        return self._balanced_selection(files, budget)
    elif strategy == "diverse":
        return self._diverse_selection(files, budget)
    else:
        return self._balanced_selection(files, budget)

estimate_tokens_for_git

Python

estimate_tokens_for_git(git_context: Optional[Dict[str, Any]]) -> int

Estimate tokens needed for git context.

Source code in tenets/core/distiller/optimizer.py

Python

def estimate_tokens_for_git(self, git_context: Optional[Dict[str, Any]]) -> int:
    """Estimate tokens needed for git context."""
    if git_context is None:
        return 0

    # Empty dict still incurs base overhead per tests
    tokens = 100  # Base overhead

    if "recent_commits" in git_context:
        # ~50 tokens per commit
        tokens += len(git_context["recent_commits"]) * 50

    if "contributors" in git_context:
        # ~20 tokens per contributor
        tokens += len(git_context["contributors"]) * 20

    if "recent_changes" in git_context:
        # ~30 tokens per file change entry
        tokens += len(git_context.get("recent_changes", [])) * 30

    return tokens

estimate_tokens_for_tenets

Python

estimate_tokens_for_tenets(tenet_count: int, with_reinforcement: bool = False) -> int

Estimate tokens needed for tenet injection.

Source code in tenets/core/distiller/optimizer.py

Python

def estimate_tokens_for_tenets(self, tenet_count: int, with_reinforcement: bool = False) -> int:
    """Estimate tokens needed for tenet injection."""
    # ~30 tokens per tenet with formatting
    tokens = tenet_count * 30

    # Reinforcement section adds ~100 tokens
    if with_reinforcement and tenet_count > 3:
        tokens += 100

    return tokens

Functions