Skip to content

summarizer

Full name: tenets.core.summarizer.summarizer

summarizer

Main summarizer orchestrator for content compression.

This module provides the main Summarizer class that coordinates different summarization strategies to compress code, documentation, and other text content while preserving important information.

The summarizer supports multiple strategies: - Extractive: Selects important sentences - Compressive: Removes redundant content - TextRank: Graph-based ranking - Transformer: Neural summarization (requires ML) - LLM: Large language model summarization (costs $)

Classes

SummarizationMode

Bases: Enum

Available summarization modes.

SummarizationResultdataclass

Python
SummarizationResult(
    original_text: str,
    summary: str,
    original_length: int,
    summary_length: int,
    compression_ratio: float,
    strategy_used: str,
    time_elapsed: float,
    metadata: Dict[str, Any] = None,
)

Result from summarization operation.

ATTRIBUTEDESCRIPTION
original_text

Original text

TYPE:str

summary

Summarized text

TYPE:str

original_length

Original text length

TYPE:int

summary_length

Summary length

TYPE:int

compression_ratio

Actual compression ratio achieved

TYPE:float

strategy_used

Which strategy was used

TYPE:str

time_elapsed

Time taken to summarize

TYPE:float

metadata

Additional metadata

TYPE:Dict[str, Any]

Attributes
reduction_percentproperty
Python
reduction_percent: float

Get reduction percentage.

Methods:
to_dict
Python
to_dict() -> Dict[str, Any]

Convert to dictionary.

Source code in tenets/core/summarizer/summarizer.py
Python
def to_dict(self) -> Dict[str, Any]:
    """Convert to dictionary."""
    return {
        "summary": self.summary,
        "original_length": self.original_length,
        "summary_length": self.summary_length,
        "compression_ratio": self.compression_ratio,
        "reduction_percent": self.reduction_percent,
        "strategy_used": self.strategy_used,
        "time_elapsed": self.time_elapsed,
        "metadata": self.metadata or {},
    }

BatchSummarizationResultdataclass

Python
BatchSummarizationResult(
    results: List[SummarizationResult],
    total_original_length: int,
    total_summary_length: int,
    overall_compression_ratio: float,
    total_time_elapsed: float,
    files_processed: int,
    files_failed: int,
)

Result from batch summarization.

Methods:
to_dict
Python
to_dict() -> Dict[str, Any]

Convert to dictionary.

Source code in tenets/core/summarizer/summarizer.py
Python
def to_dict(self) -> Dict[str, Any]:
    """Convert to dictionary."""
    return {
        "total_original_length": self.total_original_length,
        "total_summary_length": self.total_summary_length,
        "overall_compression_ratio": self.overall_compression_ratio,
        "total_time_elapsed": self.total_time_elapsed,
        "files_processed": self.files_processed,
        "files_failed": self.files_failed,
        "reduction_percent": (1 - self.overall_compression_ratio) * 100,
    }

Summarizer

Python
Summarizer(
    config: Optional[TenetsConfig] = None,
    default_mode: Optional[str] = None,
    enable_cache: bool = True,
)

Main summarization orchestrator.

Coordinates different summarization strategies and provides a unified interface for content compression. Supports single and batch processing, strategy selection, and caching.

ATTRIBUTEDESCRIPTION
config

TenetsConfig instance

logger

Logger instance

strategies

Available summarization strategies

TYPE:Dict[SummarizationMode, SummarizationStrategy]

cache

Summary cache for repeated content

TYPE:Dict[str, SummarizationResult]

stats

Summarization statistics

Initialize summarizer.

PARAMETERDESCRIPTION
config

Tenets configuration

TYPE:Optional[TenetsConfig]DEFAULT:None

default_mode

Default summarization mode

TYPE:Optional[str]DEFAULT:None

enable_cache

Whether to enable caching

TYPE:boolDEFAULT:True

Source code in tenets/core/summarizer/summarizer.py
Python
def __init__(
    self,
    config: Optional[TenetsConfig] = None,
    default_mode: Optional[str] = None,
    enable_cache: bool = True,
):
    """Initialize summarizer.

    Args:
        config: Tenets configuration
        default_mode: Default summarization mode
        enable_cache: Whether to enable caching
    """
    self.config = config or TenetsConfig()
    self.logger = get_logger(__name__)

    # Determine default mode
    if default_mode:
        self.default_mode = SummarizationMode(default_mode)
    else:
        self.default_mode = SummarizationMode.AUTO

    # Initialize strategies
    self.strategies: Dict[SummarizationMode, SummarizationStrategy] = {
        SummarizationMode.EXTRACTIVE: ExtractiveStrategy(),
        SummarizationMode.COMPRESSIVE: CompressiveStrategy(),
    }

    # Try to initialize TextRank (requires scikit-learn)
    try:
        self.strategies[SummarizationMode.TEXTRANK] = TextRankStrategy()
    except ImportError:
        self.logger.debug("TextRank unavailable (scikit-learn not installed)")

    # Try to initialize ML strategies
    self._init_ml_strategies()

    # Cache for summaries
    self.enable_cache = enable_cache
    self.cache: Dict[str, SummarizationResult] = {}

    # Statistics
    self.stats = {
        "total_summarized": 0,
        "total_time": 0.0,
        "cache_hits": 0,
        "cache_misses": 0,
        "strategies_used": {},
    }

    self.logger.info(
        f"Summarizer initialized with mode={self.default_mode.value}, "
        f"strategies={list(self.strategies.keys())}"
    )
Methods:
summarize
Python
summarize(
    text: str,
    mode: Optional[Union[str, SummarizationMode]] = None,
    target_ratio: float = 0.3,
    max_length: Optional[int] = None,
    min_length: Optional[int] = None,
    force_strategy: Optional[SummarizationStrategy] = None,
) -> SummarizationResult

Summarize text content.

PARAMETERDESCRIPTION
text

Text to summarize

TYPE:str

mode

Summarization mode (uses default if None)

TYPE:Optional[Union[str, SummarizationMode]]DEFAULT:None

target_ratio

Target compression ratio (0.3 = 30% of original)

TYPE:floatDEFAULT:0.3

max_length

Maximum summary length in characters

TYPE:Optional[int]DEFAULT:None

min_length

Minimum summary length in characters

TYPE:Optional[int]DEFAULT:None

force_strategy

Force specific strategy instance

TYPE:Optional[SummarizationStrategy]DEFAULT:None

RETURNSDESCRIPTION
SummarizationResult

SummarizationResult with summary and metadata

Example

summarizer = Summarizer() result = summarizer.summarize( ... long_text, ... mode="extractive", ... target_ratio=0.25 ... ) print(f"Reduced by {result.reduction_percent:.1f}%")

Source code in tenets/core/summarizer/summarizer.py
Python
def summarize(
    self,
    text: str,
    mode: Optional[Union[str, SummarizationMode]] = None,
    target_ratio: float = 0.3,
    max_length: Optional[int] = None,
    min_length: Optional[int] = None,
    force_strategy: Optional[SummarizationStrategy] = None,
) -> SummarizationResult:
    """Summarize text content.

    Args:
        text: Text to summarize
        mode: Summarization mode (uses default if None)
        target_ratio: Target compression ratio (0.3 = 30% of original)
        max_length: Maximum summary length in characters
        min_length: Minimum summary length in characters
        force_strategy: Force specific strategy instance

    Returns:
        SummarizationResult with summary and metadata

    Example:
        >>> summarizer = Summarizer()
        >>> result = summarizer.summarize(
        ...     long_text,
        ...     mode="extractive",
        ...     target_ratio=0.25
        ... )
        >>> print(f"Reduced by {result.reduction_percent:.1f}%")
    """
    if not text:
        return SummarizationResult(
            original_text="",
            summary="",
            original_length=0,
            summary_length=0,
            compression_ratio=1.0,
            strategy_used="none",
            time_elapsed=0.0,
        )

    start_time = time.time()

    # Check cache
    if self.enable_cache:
        cache_key = self._get_cache_key(text, target_ratio, max_length, min_length)
        if cache_key in self.cache:
            self.stats["cache_hits"] += 1
            self.logger.debug("Cache hit for summary")
            return self.cache[cache_key]
        else:
            self.stats["cache_misses"] += 1

    # Select strategy
    if force_strategy:
        strategy = force_strategy
        strategy_name = getattr(strategy, "name", "custom")
    else:
        strategy, strategy_name = self._select_strategy(text, mode, target_ratio)

    if not strategy:
        # Fallback to extractive
        strategy = self.strategies[SummarizationMode.EXTRACTIVE]
        strategy_name = "extractive"

    self.logger.debug(f"Using {strategy_name} strategy for summarization")

    # Perform summarization
    try:
        summary = strategy.summarize(
            text, target_ratio=target_ratio, max_length=max_length, min_length=min_length
        )
    except Exception as e:
        self.logger.error(f"Summarization failed with {strategy_name}: {e}")
        # Fallback to simple truncation
        summary = self._simple_truncate(text, target_ratio, max_length)
        strategy_name = "truncate"

    # Enforce min_length: if requested min_length exceeds original, do not make it shorter
    if min_length and min_length > len(text) and len(text) > 0 and len(summary) < len(text):
        summary = text

    # Create result
    result = SummarizationResult(
        original_text=text,
        summary=summary,
        original_length=len(text),
        summary_length=len(summary),
        compression_ratio=len(summary) / len(text) if text else 1.0,
        strategy_used=strategy_name,
        time_elapsed=time.time() - start_time,
        metadata={
            "target_ratio": target_ratio,
            "max_length": max_length,
            "min_length": min_length,
        },
    )

    # Update statistics
    self.stats["total_summarized"] += 1
    self.stats["total_time"] += result.time_elapsed
    self.stats["strategies_used"][strategy_name] = (
        self.stats["strategies_used"].get(strategy_name, 0) + 1
    )

    # Cache result
    if self.enable_cache:
        self.cache[cache_key] = result

    self.logger.info(
        f"Summarized {result.original_length} chars to {result.summary_length} chars "
        f"({result.reduction_percent:.1f}% reduction) using {strategy_name}"
    )

    return result
summarize_file
Python
summarize_file(
    file: FileAnalysis,
    mode: Optional[Union[str, SummarizationMode]] = None,
    target_ratio: float = 0.3,
    preserve_structure: bool = True,
    prompt_keywords: Optional[List[str]] = None,
) -> SummarizationResult

Summarize a code file intelligently.

Handles code files specially by preserving important elements like class/function signatures while summarizing implementations. Enhanced with context-aware documentation summarization that preserves relevant sections based on prompt keywords.

PARAMETERDESCRIPTION
file

FileAnalysis object

TYPE:FileAnalysis

mode

Summarization mode

TYPE:Optional[Union[str, SummarizationMode]]DEFAULT:None

target_ratio

Target compression ratio

TYPE:floatDEFAULT:0.3

preserve_structure

Whether to preserve code structure

TYPE:boolDEFAULT:True

prompt_keywords

Keywords from user prompt for context-aware summarization

TYPE:Optional[List[str]]DEFAULT:None

RETURNSDESCRIPTION
SummarizationResult

SummarizationResult

Source code in tenets/core/summarizer/summarizer.py
Python
def summarize_file(
    self,
    file: FileAnalysis,
    mode: Optional[Union[str, SummarizationMode]] = None,
    target_ratio: float = 0.3,
    preserve_structure: bool = True,
    prompt_keywords: Optional[List[str]] = None,
) -> SummarizationResult:
    """Summarize a code file intelligently.

    Handles code files specially by preserving important elements
    like class/function signatures while summarizing implementations.
    Enhanced with context-aware documentation summarization that preserves
    relevant sections based on prompt keywords.

    Args:
        file: FileAnalysis object
        mode: Summarization mode
        target_ratio: Target compression ratio
        preserve_structure: Whether to preserve code structure
        prompt_keywords: Keywords from user prompt for context-aware summarization

    Returns:
        SummarizationResult
    """
    if not file.content:
        return SummarizationResult(
            original_text="",
            summary="",
            original_length=0,
            summary_length=0,
            compression_ratio=1.0,
            strategy_used="none",
            time_elapsed=0.0,
        )

    # Determine if this is a documentation file
    file_path = Path(file.path)
    is_documentation = self._is_documentation_file(file_path)

    # Check if context-aware documentation summarization is enabled
    docs_context_aware = getattr(self.config.summarizer, "docs_context_aware", True)
    docs_show_in_place_context = getattr(
        self.config.summarizer, "docs_show_in_place_context", True
    )

    # Apply documentation-specific summarization if enabled and applicable
    if (
        is_documentation
        and docs_context_aware
        and docs_show_in_place_context
        and prompt_keywords
    ):
        summary = self._summarize_documentation_with_context(
            file, target_ratio, prompt_keywords
        )

        return SummarizationResult(
            original_text=file.content,
            summary=summary,
            original_length=len(file.content),
            summary_length=len(summary),
            compression_ratio=len(summary) / len(file.content),
            strategy_used="docs-context-aware",
            time_elapsed=0.0,
            metadata={
                "file": file.path,
                "is_documentation": True,
                "prompt_keywords": prompt_keywords,
                "context_aware": True,
            },
        )
    elif preserve_structure and file.language and not is_documentation:
        # Intelligent code summarization (skip for documentation files)
        summary = self._summarize_code(file, target_ratio)

        return SummarizationResult(
            original_text=file.content,
            summary=summary,
            original_length=len(file.content),
            summary_length=len(summary),
            compression_ratio=len(summary) / len(file.content),
            strategy_used="code-aware",
            time_elapsed=0.0,
            metadata={"file": file.path, "language": file.language},
        )
    else:
        # Regular text summarization
        return self.summarize(file.content, mode, target_ratio)
batch_summarize
Python
batch_summarize(
    texts: List[Union[str, FileAnalysis]],
    mode: Optional[Union[str, SummarizationMode]] = None,
    target_ratio: float = 0.3,
    parallel: bool = True,
    prompt_keywords: Optional[List[str]] = None,
) -> BatchSummarizationResult

Summarize multiple texts in batch.

PARAMETERDESCRIPTION
texts

List of texts or FileAnalysis objects

TYPE:List[Union[str, FileAnalysis]]

mode

Summarization mode

TYPE:Optional[Union[str, SummarizationMode]]DEFAULT:None

target_ratio

Target compression ratio

TYPE:floatDEFAULT:0.3

parallel

Whether to process in parallel

TYPE:boolDEFAULT:True

prompt_keywords

Keywords from user prompt for context-aware documentation summarization

TYPE:Optional[List[str]]DEFAULT:None

RETURNSDESCRIPTION
BatchSummarizationResult

BatchSummarizationResult

Source code in tenets/core/summarizer/summarizer.py
Python
def batch_summarize(
    self,
    texts: List[Union[str, FileAnalysis]],
    mode: Optional[Union[str, SummarizationMode]] = None,
    target_ratio: float = 0.3,
    parallel: bool = True,
    prompt_keywords: Optional[List[str]] = None,
) -> BatchSummarizationResult:
    """Summarize multiple texts in batch.

    Args:
        texts: List of texts or FileAnalysis objects
        mode: Summarization mode
        target_ratio: Target compression ratio
        parallel: Whether to process in parallel
        prompt_keywords: Keywords from user prompt for context-aware documentation summarization

    Returns:
        BatchSummarizationResult
    """
    start_time = time.time()
    results = []

    total_original = 0
    total_summary = 0
    files_failed = 0

    for item in texts:
        try:
            if isinstance(item, FileAnalysis):
                result = self.summarize_file(
                    item, mode, target_ratio, prompt_keywords=prompt_keywords
                )
            else:
                result = self.summarize(item, mode, target_ratio)

            results.append(result)
            total_original += result.original_length
            total_summary += result.summary_length

        except Exception as e:
            self.logger.error(f"Failed to summarize item: {e}")
            files_failed += 1

    overall_ratio = total_summary / total_original if total_original > 0 else 1.0

    return BatchSummarizationResult(
        results=results,
        total_original_length=total_original,
        total_summary_length=total_summary,
        overall_compression_ratio=overall_ratio,
        total_time_elapsed=time.time() - start_time,
        files_processed=len(results),
        files_failed=files_failed,
    )
clear_cache
Python
clear_cache()

Clear the summary cache.

Source code in tenets/core/summarizer/summarizer.py
Python
def clear_cache(self):
    """Clear the summary cache."""
    self.cache.clear()
    self.logger.info("Summary cache cleared")
get_stats
Python
get_stats() -> Dict[str, Any]

Get summarization statistics.

RETURNSDESCRIPTION
Dict[str, Any]

Dictionary of statistics

Source code in tenets/core/summarizer/summarizer.py
Python
def get_stats(self) -> Dict[str, Any]:
    """Get summarization statistics.

    Returns:
        Dictionary of statistics
    """
    stats = self.stats.copy()

    # Add cache stats
    stats["cache_size"] = len(self.cache)
    if self.stats["cache_hits"] + self.stats["cache_misses"] > 0:
        stats["cache_hit_rate"] = self.stats["cache_hits"] / (
            self.stats["cache_hits"] + self.stats["cache_misses"]
        )
    else:
        stats["cache_hit_rate"] = 0.0

    # Add average time
    if self.stats["total_summarized"] > 0:
        stats["avg_time"] = self.stats["total_time"] / self.stats["total_summarized"]
    else:
        stats["avg_time"] = 0.0

    return stats

FileSummarizer

Python
FileSummarizer(model: Optional[str] = None)

Backward-compatible file summarizer used by tests.

This class now delegates to the main Summarizer to avoid code duplication, while maintaining the same API expected by tests.

Source code in tenets/core/summarizer/summarizer.py
Python
def __init__(self, model: Optional[str] = None):
    self.model = model
    self.logger = get_logger(__name__)
    # Create a main summarizer instance for delegation
    config = TenetsConfig()
    self.summarizer = Summarizer(config)
Methods:
summarize_file
Python
summarize_file(path: Union[str, Path], max_lines: int = 50)

Summarize a file from disk into a FileSummary.

PARAMETERDESCRIPTION
path

Path to the file

TYPE:Union[str, Path]

max_lines

Maximum number of lines in the summary

TYPE:intDEFAULT:50

RETURNSDESCRIPTION
FileSummary

summary object with metadata

Source code in tenets/core/summarizer/summarizer.py
Python
def summarize_file(self, path: Union[str, Path], max_lines: int = 50):
    """Summarize a file from disk into a FileSummary.

    Args:
        path: Path to the file
        max_lines: Maximum number of lines in the summary

    Returns:
        FileSummary: summary object with metadata
    """
    from tenets.models.summary import FileSummary  # local import to avoid cycles

    p = Path(path)
    text = self._read_text(p)

    # Delegate to shared utilities for summary extraction
    summary_text = self._extract_summary(text, max_lines=max_lines, file_path=p)

    tokens = count_tokens(summary_text, model=self.model)
    metadata = {"strategy": "heuristic", "max_lines": max_lines}

    return FileSummary(
        path=str(p),
        summary=summary_text,
        token_count=tokens,
        metadata=metadata,
    )

Functions: