ffbridge-postmortem

FFBridge Postmortem Analyzer - Bridge Tournament Data Extraction and Analysis

Overview

This project provides a comprehensive toolkit for extracting, analyzing, and post-mortem analysis of bridge tournament data from the French Bridge Federation (FFBridge) system. It includes web scraping capabilities, data processing tools, and interactive analysis interfaces.

Key Features

• Web Scraping: Automated extraction of bridge tournament data from FFBridge and BridgePlus websites
• Data Processing: Convert raw tournament data into structured formats for analysis
• Interactive Analysis: Streamlit-based web interface for post-mortem game analysis
• Authentication: Automated bearer token management for FFBridge API access
• Bridge Library: Comprehensive library for bridge deal analysis and augmentation
• Enhanced HTML Parsing: Robust Unicode-aware parsing with improved French-to-English translation
• Type Safety: Comprehensive type definitions and static analysis support

Main Components

1. Data Extraction (mlBridgeLib/mlBridgeBPLib.py)

• Tournament discovery and club extraction
• Team and board results scraping
• Individual board/deal extraction with PBN format
• Parallel processing for high-performance data extraction
• All async functions follow _async naming convention
• Enhanced HTML parsing with dynamic board number extraction
• Robust opponent information extraction using specific HTML structure
• NEW: Improved Unicode handling for French suit symbols (♠♥♦♣)
• NEW: Enhanced contract extraction with comprehensive French-to-English strain mapping
• NEW: Robust vulnerability translation with error handling

2. Interactive Analysis (ffbridge_streamlit.py)

• Web-based interface for game analysis
• Chat-based postmortem analysis with AI assistance
• Support for both simultaneous and RRN tournaments
• Real-time data visualization and statistics
• Sidebar AI Predictions scores the open session and shows club rank, our predicted global rank, FFBridge's published global rank, and the difference

Prediction API

The postmortem API on port 8517 (python ffbridge_postmortem_api_server.py) exposes the same predictions the sidebar uses.

• GET /postmortems/{player_id}/predictions/boards — every table on every board: contract, predicted contract, declarer, predicted declarer, North-South percentage, predicted percentage, and Pct_Diff. Board_Tables is the tables that played the board, Contract_Pred_Tables how many of them played that row's predicted contract (doubles included, so 4SXW is not 4SW), and Declarer_Direction_Pred_Tables how many declared from the predicted seat (a predicted pass-out counts the passed-out tables). A miss where 8 of 13 tables played the predicted contract is the table departing from the room, not a model error.
• GET /postmortems/{player_id}/predictions/rankings — one row per pair at the player's club: Pair, Orientation, Club_Rank, Club_Rank_Pred, Global_Scratch_Rank (FFBridge's published global rank), Global_Scratch_Rank_Pred, Rank_Diff, Global_Scratch_Pct (published session percentage), Global_Scratch_Pct_Pred, Pct_Diff, Handicap_Pct, Handicap_Club_Rank, Handicap_Global_Rank, and Boards. The other clubs in the simultaneous are used only to place each prediction in the national field.

The predictions are scratch, so Club_Rank, Global_Scratch_Rank, Global_Scratch_Pct, and the diffs are scratch too. In a handicapped session (Octopus, Roy René first Tuesday) FFBridge's sessionScore and rank include the IV bonus totalBonus; the scratch percentage is sessionScore - totalBonus, and the scratch global rank places it among the whole national field. FFBridge's handicapped figures are in the Handicap_* columns, which are null in a session without handicap.

The national ranking is cached under rankings/{session_id}.json. Before FFBridge publishes a session, its ranking lists every pair at 0% and 1st place. A cached ranking like that is downloaded again on the next request and replaced once the scores arrive. Until then, the actual columns are null and the response's warning says the results are not published.

Optional query parameters are session_id and limit (default 500, maximum 2000). Omit session_id to use the latest cached session for that player. DuckDB GET /postmortems/{player_id}/sql attaches the same columns when the query mentions them (Pct_NS_Pred, Contract_Pred, Global_Scratch_Rank, Global_Scratch_Rank_Pred, and the other prediction markers).

The MCP tools ffbridge_postmortem_prediction_boards and ffbridge_postmortem_prediction_rankings call those routes. Checkpoints live in FFBRIDGE_SAVED_MODELS or E:\bridge\data\ffbridge\SavedModels. When the checkpoints are missing, published club rank and FFBridge global rank are still returned and the prediction columns are null.

Group run

A brief, full, or predictions postmortem runs as a job. The server loads the session once and runs every favorite, the same way the website does. Do not call GET /postmortems/{player_id}/sql once per favorite.

MCP clients do not call these routes themselves. They call ffbridge_postmortem_report_start, then ffbridge_postmortem_report_status, then ffbridge_postmortem_report_result, and ffbridge_postmortem_report_pdf for the PDF. The MCP report argument is Brief postmortem, Full postmortem, or Predictions, which map to the groups Summarize_Briefly, Summarize, and AI_Predictions. Omit session_id on the MCP start tool for the player's latest published game.

POST /postmortems/{player_id}/run-group returns a job_id before the SQL runs. JSON body:

• session_id (required), group (required button id), statements (required list of {id, title, sql})
• group_alias (optional display name), force (optional; default false), limit (optional row cap, max 2000)

The response is job_id, status (queued), cached, player_id, session_id, group, done, total, and elapsed_s. The same player, session, group, and SQL returns the finished job with cached: true and does not run again. force: true starts a new job.

GET /run-group/{job_id} is the poll. status is queued, running, done, or failed. done and total count favorites. elapsed_s is seconds since the job started. completed lists each finished favorite with title, elapsed_s, row_count, and error. Poll until done or failed. A client can read results as soon as completed is non-empty.

GET /run-group/{job_id}/result returns finished favorites in group order, including while status is still running. complete is true once the job is done or failed. Query offset and limit page by favorite, not by row. Query title returns one favorite (title or id, case-insensitive). Each section has columns, rows, row_count, and elapsed_s. next_offset is the next page, or null when no further finished favorites are waiting. results_file on a finished job is group_runs/<job_id>.json under the postmortem cache. That path is on the server. Clients page with the result route instead of opening the file.

Board-level statements should select Result (integer overtricks) as well as the score. The MCP start tool inserts Result into those statements. Aggregate statements that use GROUP BY are left unchanged.

Every diff is actual minus predicted. Rank_Diff is Global_Scratch_Rank - Global_Scratch_Rank_Pred; a positive value means the pair finished lower than predicted, because a smaller rank is better. Pct_Diff on a board is Pct_NS - Pct_NS_Pred, and on the rankings it is Global_Scratch_Pct - Global_Scratch_Pct_Pred; a positive value means the pair scored above the prediction. Global_Scratch_Rank_Pred places the pair's predicted percentage among the other pairs' published percentages in the national field, while Club_Rank_Pred ranks predictions against the club's predictions.

3. Authentication (ffbridge_auth_playwright.py)

• Automated bearer token extraction using Playwright
• Environment variable management for credentials
• Support for both Lancelot and EASI token types

4. Bridge Analysis Library (mlBridgeLib/)

• Deal analysis and augmentation tools
• Double dummy analysis integration
• Bidding and play analysis
• Machine learning utilities for bridge AI

Quick Start

Installation

# Install core dependencies
pip install -r requirements.txt

# Install Playwright browsers
playwright install chromium

# Optional: Install advanced features (machine learning, AI, etc.)
pip install -r requirements-optional.txt

# Set up environment variables
cp env_template.txt .env
# Edit .env with your FFBridge credentials

# Run tests to verify installation
python test_type_definitions.py
python test_bridge_library.py

# Optional: Run type checking
mypy ffbridge_streamlit.py

Basic Usage

import asyncio
from mlBridgeLib.mlBridgeBPLib import get_all_boards_for_team_async

async def extract_tournament_data():
    # Extract all boards for a team
    result = await get_all_boards_for_team_async(
        tr="S202602",  # Tournament ID
        cl="5802079",  # Club ID
        sc="A",        # Section
        eq="212"       # Team number
    )
    
    boards_df = result['boards']
    frequency_df = result['score_frequency']
    
    print(f"Extracted {len(boards_df)} boards")
    print(f"Opponent info: {boards_df['Opponent_Pair_Names'][0]} vs {boards_df['Team_Name'][0]}")
    return result

# Run extraction
data = asyncio.run(extract_tournament_data())

Interactive Analysis

# Launch Streamlit interface
streamlit run ffbridge_streamlit.py

Historical Parquet archive

Generated Lancelot postmortems are also stored as canonical, player-neutral session fragments. By default the durable archive is cache/archive; set FFBRIDGE_POSTMORTEM_ARCHIVE_DIR to place it on another durable volume.

Audit source coverage without writing:

python build_ffbridge_postmortem_archive.py --audit-only `
    --source-dir E:\bridge\data\ffbridge\data

Backfill from the authoritative ranking/team-score JSON, rebuild player/session indexes, and compact affected year/series partitions:

python build_ffbridge_postmortem_archive.py `
    --source-dir E:\bridge\data\ffbridge\data `
    --archive-dir E:\bridge\data\ffbridge\postmortem_archive

Use --limit 1 for a smoke build. Existing player cache files can be migrated with --from-cache, but raw JSON is the authoritative backfill source. The API serves archived sessions before legacy player cache files and exposes bounded cross-session projection/filter queries at GET /archive/rows. Pass --fetch-missing explicitly to download audit-reported missing ranking/team JSON before a backfill.

Measure cache duplication, archive size, projected session reads, and date aggregates with:

python benchmark_ffbridge_postmortem_archive.py `
    --cache-dir cache --archive-dir cache/archive

Automation: email a PDF report

ffbridge_postmortem_generator.py drives the live app in a headless browser, captures the generated PDF, and emails it. Combined with the included PowerShell helper this becomes a daily "email me the report for session X" job.

One-time setup:

pip install -r requirements.txt
playwright install chromium

Then put SMTP credentials in a .env file next to ffbridge_streamlit.py. Gmail with an App Password is the tested-and-working path:

SMTP_HOST=smtp.gmail.com
SMTP_PORT=587
SMTP_USER=you@gmail.com
SMTP_PASSWORD=xxxxxxxxxxxxxxxx     # 16-char Google app password, no spaces
SMTP_FROM=you@gmail.com

Important Gmail gotchas:

• SMTP_USER and SMTP_FROM must match the Google account that owns the app password. You cannot use a Yahoo (or other) address as the sender on a Gmail-authenticated connection -- Gmail will reject the message.
• 2-Step Verification must be enabled on that Google account; otherwise the "App passwords" page won't be available and any password you paste will fail with 535-5.7.8 Username and Password not accepted.
• The app password is 16 characters with no spaces (Google shows it with spaces just for readability -- strip them when saving to .env).

Send a report on demand

Email the player's most recent game right now (SMTP creds come from .env):

python ffbridge_postmortem_generator.py `
    --url   "https://ffbridge.postmortem.chat/?player_id=9500754" `
    --email coach@example.com

Pick a specific session (rarely needed — usually you want auto-latest):

python ffbridge_postmortem_generator.py `
    --url   "https://ffbridge.postmortem.chat/?player_id=9500754&session_id=107118" `
    --email coach@example.com

Force a re-send even if the cache says it already went out, and print the full SMTP conversation if something looks wrong:

python ffbridge_postmortem_generator.py `
    --url   "https://ffbridge.postmortem.chat/?player_id=9500754" `
    --email coach@example.com `
    --force-email `
    --smtp-debug

Generate the PDF locally without sending email (handy for testing the browser side in isolation):

python ffbridge_postmortem_generator.py `
    --url   "https://ffbridge.postmortem.chat/?player_id=9500754" `
    --email me@example.com `
    --output report.pdf `
    --no-email

Pass SMTP settings on the command line instead of via .env:

python ffbridge_postmortem_generator.py `
    --url        "https://ffbridge.postmortem.chat/?player_id=9500754" `
    --email      coach@example.com `
    --from-email you@gmail.com `
    --smtp-host  smtp.gmail.com `
    --smtp-port  587 `
    --smtp-user  you@gmail.com `
    --smtp-password "xxxxxxxxxxxxxxxx"

Run it daily on Windows

Register a daily 6 PM scheduled task that always picks the player's most recent game:

.\schedule_daily_report.ps1 `
    -Url   "https://ffbridge.postmortem.chat/?player_id=9500754" `
    -Email coach@example.com `
    -Time  18:00

What the scheduled run does each evening:

1. Purges any cached PDFs in report_cache\ whose mtime is older than the TTL (default 168 h / 1 week). The TTL is a sliding window: every cache hit refreshes the file's mtime, so an entry survives indefinitely as long as the player hasn't played a newer game.
2. Opens the URL in headless Chromium and waits for the report metadata.
3. Always loads the player's most recent game (no date filter).
4. Skips silently if report_cache\{player_id}_{game_date}.pdf already exists — the cache assumes the report was already emailed for that game.
5. Otherwise downloads the PDF, stores it in the cache, and emails it.

Net effect: each game produces exactly one email, no matter how often the scheduler runs in between, and no matter how many days pass between games.

Remove the scheduled task:

.\schedule_daily_report.ps1 -Unregister

Verify SMTP credentials in isolation

If a real run fails at the email step, run test_smtp.py to bisect whether the problem is in the browser/PDF side or in SMTP itself. By default it reads SMTP_* from your .env:

# Just connect, EHLO/STARTTLS, and authenticate -- don't actually send:
python test_smtp.py --no-send

# Send a tiny test message to yourself:
python test_smtp.py

Dependencies

Core Dependencies

The main requirements are in requirements.txt and include:

• Streamlit: Web interface and data visualization
• Polars/Pandas: Data processing and analysis
• Playwright: Web scraping and automation
• Endplay: Bridge game analysis and double dummy calculations
• BeautifulSoup: HTML parsing
• Matplotlib/Seaborn: Data visualization
• ReportLab: PDF report generation

Optional Dependencies

Advanced features require additional packages in requirements-optional.txt:

• PyTorch/FastAI: Machine learning capabilities (mlBridgeAiLib.py)
• Scikit-learn: Machine learning utilities
• SQLAlchemy: Database support
• OpenAI: AI integration for chat features

Documentation

• Library Usage Guide: Comprehensive guide for using the bridge scraping library
• Authentication Guide: Setup and usage of automated authentication
• Test Suite: Run python test_bridge_library.py for comprehensive testing
• Type Definitions Test: Run python test_type_definitions.py for type validation
• Type Checking: Run mypy ffbridge_streamlit.py for static type analysis

Recent Updates

Version 2.6.0 (Latest)

• Streamlined Dependencies: Reorganized requirements.txt with clear separation of core vs optional dependencies
• Optional Features: Created requirements-optional.txt for advanced features like machine learning
• Better Documentation: Updated installation instructions and dependency documentation
• Cleaner Setup: Removed unused dependencies and improved version constraints

Version 2.5.0 (Previous)

• Enhanced HTML Parsing: Improved Unicode handling for French suit symbols (♠♥♦♣)
• Robust Contract Extraction: Comprehensive French-to-English strain mapping with error handling
• Improved Vulnerability Translation: Enhanced error handling for French vulnerability terms
• Better Regex Patterns: Updated regex patterns to handle Unicode characters and complex HTML structures
• Type Safety Improvements: Enhanced type hints and validation throughout the codebase
• Documentation Updates: Comprehensive updates to reflect latest capabilities

Version 2.4.0 (Previous)

• Comprehensive Documentation: Created detailed LIBRARY_USAGE_GUIDE.md with complete API documentation
• Enhanced Type Safety: Improved TypedDict definitions for API configurations and data structures
• Streamlit Interface: Added comprehensive chat-based postmortem analysis with AI assistance
• API Integration: Enhanced FFBridge API integration with dual token authentication (Lancelot + EASI)
• Error Handling: Improved error handling and validation throughout the application
• Testing: Updated test suite with comprehensive coverage for new functionality

Version 2.3.0 (Previous)

• Comprehensive Type Hints: Added full type annotations throughout the codebase for better IDE support and code quality
• Enhanced Documentation: Improved function docstrings with detailed parameter and return type descriptions
• Type Safety: Added TypedDict definitions for complex data structures and API configurations
• Development Tools: Added mypy and typing-extensions for static type checking
• Code Quality: Better organized requirements.txt with version constraints and categorized dependencies

Version 2.2.0 (Previous)

• Enhanced HTML Parsing: Dynamic board number extraction from HTML instead of hardcoded assumptions
• Improved Opponent Extraction: Uses specific HTML div structure (<div class="col">contre <span class="paires">...) for reliable opponent information
• Better Error Handling: Graduated warnings instead of hard failures, with detailed diagnostic information
• Data Correspondence: Board numbers, scores, and matchpoints are extracted from same HTML rows to ensure accuracy
• Robust Validation: Enhanced data validation with detailed troubleshooting information
• Fixed Board Parsing: Removed hardcoded board skip patterns (19, 20) that were tournament-specific

Version 2.1.0 (Previous)

• New Data Fields: Added Opponent_Pair_Section field for better opponent tracking
• Enhanced Validation: Automatic team existence checking with detailed diagnostics
• Improved Error Handling: Better error messages and graceful handling of missing data
• Section Validation: Automatic detection of opponent section mismatches

Version 2.0.0 (Previous)

• Async Naming Convention: All async functions now use _async suffix
• New Functions: Added get_board_for_player_async(), get_board_for_team(), and optimized player-specific extraction
• Enhanced Error Handling: Improved robustness in web scraping operations
• Authentication Improvements: Better token management and environment variable handling

Technical Improvements

Enhanced Data Extraction

• Dynamic Board Discovery: Automatically detects board numbers from HTML content
• Opponent Information: Comprehensive opponent tracking with direction, names, number, and section
• Data Integrity: Maintains correspondence between board numbers, scores, and matchpoints
• Error Diagnostics: Detailed troubleshooting information for common issues

Improved HTML Parsing

# OLD: Hardcoded assumptions
for i in range(1, 27):
    if i not in [19, 20]:  # Skip boards 19 and 20 (not played)
        board_nums.append(i)

# NEW: Dynamic extraction from HTML
board_data = []
data_rows = await page.query_selector_all('div.row')
for row in data_rows:
    # Extract board number, score, and matchpoints from same row
    # Ensures correspondence between related data

Enhanced French-to-English Translation

# NEW: Comprehensive strain mapping including Unicode symbols
FRENCH_TO_ENGLISH_STRAIN_MAP = {
    'P': 'S',    # Spades (Piques)
    'C': 'H',    # Hearts (Cœurs)
    'K': 'D',    # Diamonds (Carreau)
    'T': 'C',    # Clubs (Trèfle)
    'sa': 'N',   # No Trump (Sans Atout) - lowercase
    'SA': 'N',   # No Trump (Sans Atout) - uppercase
    '♠': 'S',    # Unicode spade symbol
    '♥': 'H',    # Unicode heart symbol
    '♦': 'D',    # Unicode diamond symbol
    '♣': 'C',    # Unicode club symbol
    'pique': 'S',    # French class names
    'coeur': 'H',    # French class names
    'carreau': 'D',  # French class names
    'trefle': 'C',   # French class names
}

Robust Contract Extraction

# NEW: Enhanced regex pattern for contract extraction
suit_span_match = re.search(
    r'Contrat.*?(?:<span class="(pique|coeur|carreau|trefle)">[^<]*</span>|(SA))', 
    contract_span_html
)
if suit_span_match and suit_span_match.group(1):  # Suit class found
    suit_class = suit_span_match.group(1)
elif suit_span_match and suit_span_match.group(2):  # SA match found
    suit_class = suit_span_match.group(2)
else:
    raise ValueError(f"Could not find suit in contract span: {contract_span_html}")
strain = FRENCH_TO_ENGLISH_STRAIN_MAP[suit_class]

Robust Opponent Extraction

# Uses specific HTML structure for reliable extraction
opponent_div = await page.query_selector('div.col:has-text("contre")')
if opponent_div:
    paires_span = await opponent_div.query_selector('span.paires')
    if paires_span:
        # Parse opponent text like "NS : SININGE - KRIEF (113 A)"
        paires_text = paires_span.text_content()
        # Extract direction, names, number, and section

Project Structure

ffbridge-postmortem/
├── mlBridgeLib/           # Core bridge analysis library
│   ├── mlBridgeBPLib.py   # BridgePlus scraping library (main)
│   ├── mlBridgeLib.py     # Core bridge utilities
│   ├── mlBridgeAugmentLib.py  # Deal augmentation tools
│   └── ...                # Other bridge analysis modules
├── streamlitlib/          # Streamlit utilities
├── cache/                 # Cached API responses
├── competitions/          # Tournament data storage
├── results/               # Analysis results
├── old/                   # Legacy code and documentation
├── api-*.bat             # API testing scripts
├── ffbridge_streamlit.py  # Main Streamlit application
├── test_bridge_library.py # Comprehensive test suite
├── LIBRARY_USAGE_GUIDE.md # Detailed usage documentation
└── requirements.txt       # Python dependencies

Data Schema

Board Results DataFrame

• Board, Score, Percentage, Contract, Declarer, Lead
• Pair_Names, Pair_Direction, Pair_Number, Section
• Opponent_Pair_Names, Opponent_Pair_Direction, Opponent_Pair_Number, Opponent_Pair_Section

Boards DataFrame

• Board, PBN, Top, Dealer, Vul, Contract, Result, Score
• Team_Name, Pair_Number, Section, Tournament_ID, Club_ID
• Opponent_Pair_Names, Opponent_Pair_Direction, Opponent_Pair_Number, Opponent_Pair_Section

Score Frequency DataFrame

• Board, Score, Frequency, Matchpoints_NS, Matchpoints_EW

Performance Features

• Parallel Processing: Concurrent extraction of multiple boards
• Shared Browser Context: Reuse browser instances for multiple requests
• Intelligent Waiting: Adaptive delays based on page load times
• Robust Error Recovery: Graceful handling of network issues and page changes
• Efficient Data Structures: Polars DataFrames for high-performance data operations

Code Quality

• Type Safety: Comprehensive type hints throughout the codebase using Python's typing system
• Static Analysis: Support for mypy static type checking
• Documentation: Detailed docstrings with parameter and return type descriptions
• Structured Data: TypedDict definitions for complex API configurations and data structures
• IDE Support: Enhanced autocomplete and error detection in modern IDEs

Error Handling

The library provides comprehensive error handling with detailed diagnostics:

# Enhanced error messages with suggestions
if len(route_data) < 5:
    logger.warning(f"Very few route records extracted: only {len(route_data)} records found")
    logger.warning("This might indicate:")
    logger.warning("  1. Team played few boards")
    logger.warning("  2. HTML parsing issues")
    logger.warning("  3. Page structure changed")

Contributing

This project is actively developed and maintained. Key areas for contribution:

• Additional bridge analysis algorithms
• UI/UX improvements for the Streamlit interface
• Performance optimizations for data extraction
• Support for additional tournament formats
• Enhanced error handling and diagnostics
• Type safety improvements and additional type hints
• Documentation enhancements

Testing

The project includes comprehensive test suites to ensure code quality and functionality:

Type Definitions Test

python test_type_definitions.py

Tests the core type definitions and basic functionality without external dependencies.

Bridge Library Test

python test_bridge_library.py

Comprehensive test suite for bridge library functionality, API integration, and data processing.

Test Coverage

• Type Definitions: ApiUrlConfig, ApiUrlsDict, DataFramesDict
• API Integration: Authentication, request handling, error management
• Data Processing: DataFrame operations, validation, augmentation
• Bridge Library: Tournament discovery, data extraction, board analysis
• Streamlit Interface: UI components, session state, report generation

This tests all major functions and provides detailed performance metrics.

Type Checking

For static type analysis:

# Check all main files
python check_types.py

# Check specific file
mypy ffbridge_streamlit.py

# Check with custom configuration
mypy ffbridge_streamlit.py --config-file mypy.ini

This will check for type errors and provide suggestions for improving type safety.

License

MIT License - See LICENSE file for details.

Support

For issues and questions:

1. Check the Library Usage Guide for detailed documentation
2. Run the test suite to verify installation
3. Review error messages for diagnostic information
4. Check the old/ directory for legacy documentation