summarizer¶
Full name: tenets.core.summarizer.summarizer
summarizer¶
Main summarizer orchestrator for content compression.
This module provides the main Summarizer class that coordinates different summarization strategies to compress code, documentation, and other text content while preserving important information.
The summarizer supports multiple strategies: - Extractive: Selects important sentences - Compressive: Removes redundant content - TextRank: Graph-based ranking - Transformer: Neural summarization (requires ML) - LLM: Large language model summarization (costs $)
Classes¶
SummarizationResultdataclass¶
SummarizationResult(
original_text: str,
summary: str,
original_length: int,
summary_length: int,
compression_ratio: float,
strategy_used: str,
time_elapsed: float,
metadata: Dict[str, Any] = None,
)
Result from summarization operation.
| ATTRIBUTE | DESCRIPTION |
|---|---|
original_text | Original text TYPE: |
summary | Summarized text TYPE: |
original_length | Original text length TYPE: |
summary_length | Summary length TYPE: |
compression_ratio | Actual compression ratio achieved TYPE: |
strategy_used | Which strategy was used TYPE: |
time_elapsed | Time taken to summarize TYPE: |
metadata | Additional metadata |
Attributes¶
Methods:¶
to_dict¶
Convert to dictionary.
Source code in tenets/core/summarizer/summarizer.py
def to_dict(self) -> Dict[str, Any]:
"""Convert to dictionary."""
return {
"summary": self.summary,
"original_length": self.original_length,
"summary_length": self.summary_length,
"compression_ratio": self.compression_ratio,
"reduction_percent": self.reduction_percent,
"strategy_used": self.strategy_used,
"time_elapsed": self.time_elapsed,
"metadata": self.metadata or {},
}
BatchSummarizationResultdataclass¶
BatchSummarizationResult(
results: List[SummarizationResult],
total_original_length: int,
total_summary_length: int,
overall_compression_ratio: float,
total_time_elapsed: float,
files_processed: int,
files_failed: int,
)
Result from batch summarization.
Methods:¶
to_dict¶
Convert to dictionary.
Source code in tenets/core/summarizer/summarizer.py
def to_dict(self) -> Dict[str, Any]:
"""Convert to dictionary."""
return {
"total_original_length": self.total_original_length,
"total_summary_length": self.total_summary_length,
"overall_compression_ratio": self.overall_compression_ratio,
"total_time_elapsed": self.total_time_elapsed,
"files_processed": self.files_processed,
"files_failed": self.files_failed,
"reduction_percent": (1 - self.overall_compression_ratio) * 100,
}
Summarizer¶
Summarizer(
config: Optional[TenetsConfig] = None,
default_mode: Optional[str] = None,
enable_cache: bool = True,
)
Main summarization orchestrator.
Coordinates different summarization strategies and provides a unified interface for content compression. Supports single and batch processing, strategy selection, and caching.
| ATTRIBUTE | DESCRIPTION |
|---|---|
config | TenetsConfig instance |
logger | Logger instance |
strategies | Available summarization strategies |
cache | Summary cache for repeated content TYPE: |
stats | Summarization statistics |
Initialize summarizer.
| PARAMETER | DESCRIPTION |
|---|---|
config | Tenets configuration TYPE: |
default_mode | Default summarization mode |
enable_cache | Whether to enable caching TYPE: |
Source code in tenets/core/summarizer/summarizer.py
def __init__(
self,
config: Optional[TenetsConfig] = None,
default_mode: Optional[str] = None,
enable_cache: bool = True,
):
"""Initialize summarizer.
Args:
config: Tenets configuration
default_mode: Default summarization mode
enable_cache: Whether to enable caching
"""
self.config = config or TenetsConfig()
self.logger = get_logger(__name__)
# Determine default mode
if default_mode:
self.default_mode = SummarizationMode(default_mode)
else:
self.default_mode = SummarizationMode.AUTO
# Initialize strategies
self.strategies: Dict[SummarizationMode, SummarizationStrategy] = {
SummarizationMode.EXTRACTIVE: ExtractiveStrategy(),
SummarizationMode.COMPRESSIVE: CompressiveStrategy(),
}
# Try to initialize TextRank (requires scikit-learn)
try:
self.strategies[SummarizationMode.TEXTRANK] = TextRankStrategy()
except ImportError:
self.logger.debug("TextRank unavailable (scikit-learn not installed)")
# Try to initialize ML strategies
self._init_ml_strategies()
# Cache for summaries
self.enable_cache = enable_cache
self.cache: Dict[str, SummarizationResult] = {}
# Statistics
self.stats = {
"total_summarized": 0,
"total_time": 0.0,
"cache_hits": 0,
"cache_misses": 0,
"strategies_used": {},
}
self.logger.info(
f"Summarizer initialized with mode={self.default_mode.value}, "
f"strategies={list(self.strategies.keys())}"
)
Methods:¶
summarize¶
summarize(
text: str,
mode: Optional[Union[str, SummarizationMode]] = None,
target_ratio: float = 0.3,
max_length: Optional[int] = None,
min_length: Optional[int] = None,
force_strategy: Optional[SummarizationStrategy] = None,
) -> SummarizationResult
Summarize text content.
| PARAMETER | DESCRIPTION |
|---|---|
text | Text to summarize TYPE: |
mode | Summarization mode (uses default if None) TYPE: |
target_ratio | Target compression ratio (0.3 = 30% of original) TYPE: |
max_length | Maximum summary length in characters |
min_length | Minimum summary length in characters |
force_strategy | Force specific strategy instance TYPE: |
| RETURNS | DESCRIPTION |
|---|---|
SummarizationResult | SummarizationResult with summary and metadata |
Example
summarizer = Summarizer() result = summarizer.summarize( ... long_text, ... mode="extractive", ... target_ratio=0.25 ... ) print(f"Reduced by {result.reduction_percent:.1f}%")
Source code in tenets/core/summarizer/summarizer.py
def summarize(
self,
text: str,
mode: Optional[Union[str, SummarizationMode]] = None,
target_ratio: float = 0.3,
max_length: Optional[int] = None,
min_length: Optional[int] = None,
force_strategy: Optional[SummarizationStrategy] = None,
) -> SummarizationResult:
"""Summarize text content.
Args:
text: Text to summarize
mode: Summarization mode (uses default if None)
target_ratio: Target compression ratio (0.3 = 30% of original)
max_length: Maximum summary length in characters
min_length: Minimum summary length in characters
force_strategy: Force specific strategy instance
Returns:
SummarizationResult with summary and metadata
Example:
>>> summarizer = Summarizer()
>>> result = summarizer.summarize(
... long_text,
... mode="extractive",
... target_ratio=0.25
... )
>>> print(f"Reduced by {result.reduction_percent:.1f}%")
"""
if not text:
return SummarizationResult(
original_text="",
summary="",
original_length=0,
summary_length=0,
compression_ratio=1.0,
strategy_used="none",
time_elapsed=0.0,
)
start_time = time.time()
# Check cache
if self.enable_cache:
cache_key = self._get_cache_key(text, target_ratio, max_length, min_length)
if cache_key in self.cache:
self.stats["cache_hits"] += 1
self.logger.debug("Cache hit for summary")
return self.cache[cache_key]
else:
self.stats["cache_misses"] += 1
# Select strategy
if force_strategy:
strategy = force_strategy
strategy_name = getattr(strategy, "name", "custom")
else:
strategy, strategy_name = self._select_strategy(text, mode, target_ratio)
if not strategy:
# Fallback to extractive
strategy = self.strategies[SummarizationMode.EXTRACTIVE]
strategy_name = "extractive"
self.logger.debug(f"Using {strategy_name} strategy for summarization")
# Perform summarization
try:
summary = strategy.summarize(
text, target_ratio=target_ratio, max_length=max_length, min_length=min_length
)
except Exception as e:
self.logger.error(f"Summarization failed with {strategy_name}: {e}")
# Fallback to simple truncation
summary = self._simple_truncate(text, target_ratio, max_length)
strategy_name = "truncate"
# Enforce min_length: if requested min_length exceeds original, do not make it shorter
if min_length and min_length > len(text) and len(text) > 0 and len(summary) < len(text):
summary = text
# Create result
result = SummarizationResult(
original_text=text,
summary=summary,
original_length=len(text),
summary_length=len(summary),
compression_ratio=len(summary) / len(text) if text else 1.0,
strategy_used=strategy_name,
time_elapsed=time.time() - start_time,
metadata={
"target_ratio": target_ratio,
"max_length": max_length,
"min_length": min_length,
},
)
# Update statistics
self.stats["total_summarized"] += 1
self.stats["total_time"] += result.time_elapsed
self.stats["strategies_used"][strategy_name] = (
self.stats["strategies_used"].get(strategy_name, 0) + 1
)
# Cache result
if self.enable_cache:
self.cache[cache_key] = result
self.logger.info(
f"Summarized {result.original_length} chars to {result.summary_length} chars "
f"({result.reduction_percent:.1f}% reduction) using {strategy_name}"
)
return result
summarize_file¶
summarize_file(
file: FileAnalysis,
mode: Optional[Union[str, SummarizationMode]] = None,
target_ratio: float = 0.3,
preserve_structure: bool = True,
prompt_keywords: Optional[List[str]] = None,
) -> SummarizationResult
Summarize a code file intelligently.
Handles code files specially by preserving important elements like class/function signatures while summarizing implementations. Enhanced with context-aware documentation summarization that preserves relevant sections based on prompt keywords.
| PARAMETER | DESCRIPTION |
|---|---|
file | FileAnalysis object TYPE: |
mode | Summarization mode TYPE: |
target_ratio | Target compression ratio TYPE: |
preserve_structure | Whether to preserve code structure TYPE: |
prompt_keywords | Keywords from user prompt for context-aware summarization |
| RETURNS | DESCRIPTION |
|---|---|
SummarizationResult | SummarizationResult |
Source code in tenets/core/summarizer/summarizer.py
def summarize_file(
self,
file: FileAnalysis,
mode: Optional[Union[str, SummarizationMode]] = None,
target_ratio: float = 0.3,
preserve_structure: bool = True,
prompt_keywords: Optional[List[str]] = None,
) -> SummarizationResult:
"""Summarize a code file intelligently.
Handles code files specially by preserving important elements
like class/function signatures while summarizing implementations.
Enhanced with context-aware documentation summarization that preserves
relevant sections based on prompt keywords.
Args:
file: FileAnalysis object
mode: Summarization mode
target_ratio: Target compression ratio
preserve_structure: Whether to preserve code structure
prompt_keywords: Keywords from user prompt for context-aware summarization
Returns:
SummarizationResult
"""
if not file.content:
return SummarizationResult(
original_text="",
summary="",
original_length=0,
summary_length=0,
compression_ratio=1.0,
strategy_used="none",
time_elapsed=0.0,
)
# Determine if this is a documentation file
file_path = Path(file.path)
is_documentation = self._is_documentation_file(file_path)
# Check if context-aware documentation summarization is enabled
docs_context_aware = getattr(self.config.summarizer, "docs_context_aware", True)
docs_show_in_place_context = getattr(
self.config.summarizer, "docs_show_in_place_context", True
)
# Apply documentation-specific summarization if enabled and applicable
if (
is_documentation
and docs_context_aware
and docs_show_in_place_context
and prompt_keywords
):
summary = self._summarize_documentation_with_context(
file, target_ratio, prompt_keywords
)
return SummarizationResult(
original_text=file.content,
summary=summary,
original_length=len(file.content),
summary_length=len(summary),
compression_ratio=len(summary) / len(file.content),
strategy_used="docs-context-aware",
time_elapsed=0.0,
metadata={
"file": file.path,
"is_documentation": True,
"prompt_keywords": prompt_keywords,
"context_aware": True,
},
)
elif preserve_structure and file.language and not is_documentation:
# Intelligent code summarization (skip for documentation files)
summary = self._summarize_code(file, target_ratio)
return SummarizationResult(
original_text=file.content,
summary=summary,
original_length=len(file.content),
summary_length=len(summary),
compression_ratio=len(summary) / len(file.content),
strategy_used="code-aware",
time_elapsed=0.0,
metadata={"file": file.path, "language": file.language},
)
else:
# Regular text summarization
return self.summarize(file.content, mode, target_ratio)
batch_summarize¶
batch_summarize(
texts: List[Union[str, FileAnalysis]],
mode: Optional[Union[str, SummarizationMode]] = None,
target_ratio: float = 0.3,
parallel: bool = True,
prompt_keywords: Optional[List[str]] = None,
) -> BatchSummarizationResult
Summarize multiple texts in batch.
| PARAMETER | DESCRIPTION |
|---|---|
texts | List of texts or FileAnalysis objects TYPE: |
mode | Summarization mode TYPE: |
target_ratio | Target compression ratio TYPE: |
parallel | Whether to process in parallel TYPE: |
prompt_keywords | Keywords from user prompt for context-aware documentation summarization |
| RETURNS | DESCRIPTION |
|---|---|
BatchSummarizationResult | BatchSummarizationResult |
Source code in tenets/core/summarizer/summarizer.py
def batch_summarize(
self,
texts: List[Union[str, FileAnalysis]],
mode: Optional[Union[str, SummarizationMode]] = None,
target_ratio: float = 0.3,
parallel: bool = True,
prompt_keywords: Optional[List[str]] = None,
) -> BatchSummarizationResult:
"""Summarize multiple texts in batch.
Args:
texts: List of texts or FileAnalysis objects
mode: Summarization mode
target_ratio: Target compression ratio
parallel: Whether to process in parallel
prompt_keywords: Keywords from user prompt for context-aware documentation summarization
Returns:
BatchSummarizationResult
"""
start_time = time.time()
results = []
total_original = 0
total_summary = 0
files_failed = 0
for item in texts:
try:
if isinstance(item, FileAnalysis):
result = self.summarize_file(
item, mode, target_ratio, prompt_keywords=prompt_keywords
)
else:
result = self.summarize(item, mode, target_ratio)
results.append(result)
total_original += result.original_length
total_summary += result.summary_length
except Exception as e:
self.logger.error(f"Failed to summarize item: {e}")
files_failed += 1
overall_ratio = total_summary / total_original if total_original > 0 else 1.0
return BatchSummarizationResult(
results=results,
total_original_length=total_original,
total_summary_length=total_summary,
overall_compression_ratio=overall_ratio,
total_time_elapsed=time.time() - start_time,
files_processed=len(results),
files_failed=files_failed,
)
clear_cache¶
get_stats¶
Get summarization statistics.
| RETURNS | DESCRIPTION |
|---|---|
Dict[str, Any] | Dictionary of statistics |
Source code in tenets/core/summarizer/summarizer.py
def get_stats(self) -> Dict[str, Any]:
"""Get summarization statistics.
Returns:
Dictionary of statistics
"""
stats = self.stats.copy()
# Add cache stats
stats["cache_size"] = len(self.cache)
if self.stats["cache_hits"] + self.stats["cache_misses"] > 0:
stats["cache_hit_rate"] = self.stats["cache_hits"] / (
self.stats["cache_hits"] + self.stats["cache_misses"]
)
else:
stats["cache_hit_rate"] = 0.0
# Add average time
if self.stats["total_summarized"] > 0:
stats["avg_time"] = self.stats["total_time"] / self.stats["total_summarized"]
else:
stats["avg_time"] = 0.0
return stats
FileSummarizer¶
Backward-compatible file summarizer used by tests.
This class now delegates to the main Summarizer to avoid code duplication, while maintaining the same API expected by tests.
Source code in tenets/core/summarizer/summarizer.py
Methods:¶
summarize_file¶
Summarize a file from disk into a FileSummary.
| PARAMETER | DESCRIPTION |
|---|---|
path | Path to the file |
max_lines | Maximum number of lines in the summary TYPE: |
| RETURNS | DESCRIPTION |
|---|---|
FileSummary | summary object with metadata |
Source code in tenets/core/summarizer/summarizer.py
def summarize_file(self, path: Union[str, Path], max_lines: int = 50):
"""Summarize a file from disk into a FileSummary.
Args:
path: Path to the file
max_lines: Maximum number of lines in the summary
Returns:
FileSummary: summary object with metadata
"""
from tenets.models.summary import FileSummary # local import to avoid cycles
p = Path(path)
text = self._read_text(p)
# Delegate to shared utilities for summary extraction
summary_text = self._extract_summary(text, max_lines=max_lines, file_path=p)
tokens = count_tokens(summary_text, model=self.model)
metadata = {"strategy": "heuristic", "max_lines": max_lines}
return FileSummary(
path=str(p),
summary=summary_text,
token_count=tokens,
metadata=metadata,
)