14 Operator Improvements and Best Practices

This chapter covers advanced techniques for improving operator quality, reliability, and maintainability. You’ll learn essential practices for logging, error handling, testing, and optimization that ensure your operators work robustly in production environments.

Learning Objectives

By the end of this chapter, you will be able to: - Implement comprehensive logging and debugging strategies - Build robust error handling and input validation - Create comprehensive test suites for operators - Apply performance optimization techniques - Follow best practices for production-ready operators

Prerequisites

Before proceeding, ensure you’ve completed: - Basic Implementation chapter for core operator concepts - Advanced Features chapter for complex functionality - Data Input and Output Patterns for data handling

14.1 Logging and Debugging

Effective logging is essential for monitoring operator behavior and diagnosing issues in production environments.

14.1.1 Basic Logging

Implement logging for production operators:

ctx$log("Your message.")

ctx.log("Your message.")

Logging Best Practices

Log key milestones: Start/end of major operations
Include data metrics: Row counts, processing times, memory usage
Log parameter values: Help reproduce issues with specific inputs
Use structured formats: Enable easier log parsing and analysis
Avoid logging sensitive data: Protect user privacy and security

14.2 Error Handling

Implement comprehensive error handling that provides helpful feedback to users:

# Comprehensive error handling with user-friendly messages
robust_operator <- function(ctx) {
  tryCatch({
    # Validate inputs first
    validate_inputs(ctx)
    
    # Main processing with progress logging
    ctx$log(paste("[INFO]", Sys.time(), "- Beginning data analysis"))
    
    # Check for edge cases
    data <- ctx$select(.ri, .ci, .y)
    
    if (any(is.infinite(data$.y))) {
      ctx$log(paste("[WARNING]", Sys.time(), "- Infinite values detected, removing them"))
      data <- data[is.finite(data$.y), ]
    }
    
    if (nrow(data) == 0) {
      stop("No valid data remaining after cleaning")
    }
    
    # Perform analysis
    result <- perform_analysis(data)
    
    ctx$log(paste("[INFO]", Sys.time(), "- Analysis completed successfully"))
    return(result)
    
  }, error = function(e) {
    # Log the technical error
    ctx$log(paste("[ERROR]", Sys.time(), "- Technical error:", e$message))
    
    # Provide user-friendly error message
    if (grepl("projection.*required", e$message)) {
      stop("Please ensure you have dragged the required data columns to the appropriate axes.")
    } else if (grepl("data points required", e$message)) {
      stop("This analysis requires at least 3 data points. Please check your data selection.")
    } else if (grepl("values must vary", e$message)) {
      stop("The data values do not vary enough for this analysis. Please check your input data.")
    } else {
      stop(paste("An error occurred during analysis:", e$message))
    }
  })
}

def robust_operator(tercen_ctx):
    """Operator with comprehensive error handling"""
    
    try:
        # Validate inputs first
        validate_inputs(tercen_ctx)
        
        # Main processing with progress logging
        tercen_ctx.log(f"[INFO] {datetime.now()} - Beginning data analysis")
        
        # Check for edge cases
        df = tercen_ctx.select(['.ri', '.ci', '.y'], df_lib="polars")
        
        # Handle infinite values
        infinite_count = df.filter(pl.col('.y').is_infinite()).height
        if infinite_count > 0:
            tercen_ctx.log(f"[WARNING] {datetime.now()} - Infinite values detected ({infinite_count}), removing them")
            df = df.filter(pl.col('.y').is_finite())
        
        if len(df) == 0:
            raise ValueError("No valid data remaining after cleaning")
        
        # Perform analysis
        result = perform_analysis(df)
        
        tercen_ctx.log(f"[INFO] {datetime.now()} - Analysis completed successfully")
        return result
        
    except ValueError as ve:
        tercen_ctx.log(f"[ERROR] {datetime.now()} - Validation error: {str(ve)}")
        
        # Provide user-friendly error messages
        if "projection" in str(ve) and "required" in str(ve):
            raise ValueError("Please ensure you have dragged the required data columns to the appropriate axes.")
        elif "data points required" in str(ve):
            raise ValueError("This analysis requires at least 3 data points. Please check your data selection.")
        elif "values must vary" in str(ve):
            raise ValueError("The data values do not vary enough for this analysis. Please check your input data.")
        else:
            raise ValueError(f"An error occurred during analysis: {str(ve)}")
            
    except Exception as e:
        tercen_ctx.log(f"[ERROR] {datetime.now()} - Unexpected error: {str(e)}")
        raise ValueError(f"An unexpected error occurred. Please check your data and try again. Error: {str(e)}")

Error Handling Guidelines

Validate early: Check inputs before expensive computations
Fail gracefully: Provide clear, actionable error messages
Log technical details: Help with debugging while keeping user messages simple
Handle edge cases: Account for missing data, infinite values, empty datasets
Test error scenarios: Ensure error handling works as expected

14.3 Memory: tell Tercen what your operator needs

When Tercen runs your operator it puts the container under a hard memory limit. If the operator doesn’t declare its needs, the limit defaults to about 500 MiB. An operator whose real usage sits near or above that will fail intermittently with:

Failed to run operator, please increase memory resources.   (exit code 137)

“Intermittently” is the tell: the run sits exactly on its ceiling, and whether the kernel can reclaim enough cache decides its fate. Same data, different outcome.

14.3.1 The quick fix (no release needed)

On the failing workflow step, add the environment setting ram with the value in bytes (for example 2000000000 for 2 GB), reset the step and rerun. A user-defined value is always respected verbatim.

14.3.2 The durable fix: `memory_model.json`

Ship a memory_model.json at the repository root. Tercen reads it at install/update time and books memory per run based on the projected data:

{
  "kind": "OperatorEstimateModel",
  "intercept": 1.0,
  "offset": 500.0,
  "features": [
    { "kind": "Feature", "name": "n_main", "coefficient": 0.02, "exponent": 1.0 }
  ]
}

The estimate, in MB, is intercept × Π(coefficient × feature^exponent) + 1.5 × offset — the example books 750 MB plus 0.02 MB per projected cell.

Available features:

Feature	Value
`n_main`	number of projected crosstab cells
`n_cols`	rows of the column table
`n_rows`	rows of the row table
`settings.<Name>`	numeric value of an operator setting
`a_\|\|_b`	interaction: product of two features

Warning

Prefer the schema features. A settings.* feature whose setting is left blank evaluates to 0, which can silently disable the whole model (an infinite or zero estimate is discarded).

14.3.3 How the formula is evaluated — worked example

The model is multiplicative in the features, plus a constant:

estimate_MB = intercept × ∏ ( coefficient_i × feature_i ^ exponent_i )  +  1.5 × offset

Take the model above and a crosstab of 200 kinases × 12 samples, so n_main = 2 400 cells:

growth   = 1.0 × (0.02 × 2400^1.0) = 48 MB
constant = 1.5 × 500               = 750 MB
booked   = 48 + 750                = 798 MB

At 100 000 cells the same model books 750 + 2 000 = 2 750 MB — the booking follows the data instead of being a fixed number.

Two structural consequences of this shape are worth internalising:

offset is the additive floor (times 1.5). It represents everything that exists before your data does: the R or Python runtime, loaded libraries, reference databases read into memory. For many operators this constant is the whole story.
intercept and the features form the growth term. With several features the terms multiply, so two linear features model an n × m surface, not a sum. If you want additive behaviour, put it in offset and keep one feature.

14.3.4 Three real models, three shapes

Constant-dominated — pamgene/cs_uka_operator. Measured production runs peaked at 525–592 MB whether the crosstab had 600 cells or 12 000: the UKA reference database and the permutation machinery dominate; the projected data is a rounding error. The model is therefore mostly floor, with a token slope as insurance:

{
  "kind": "OperatorEstimateModel",
  "intercept": 1.0,
  "offset": 500.0,
  "features": [
    { "kind": "Feature", "name": "n_main", "coefficient": 0.02, "exponent": 1.0 }
  ]
}

750 MB + 0.02 MB/cell — fitted against 203 recorded runs, covering every one with 1.3–1.9× headroom. (Before this model existed, the operator got the 500 MiB default: every run sat exactly on its ceiling, and the intermittent exit-137 reports rolled in for months.)

Linear in the data — tercen/plot_operator. A plot’s memory scales with the points drawn, on top of an R baseline:

{
  "kind": "OperatorEstimateModel",
  "intercept": 0.0008,
  "offset": 1000.0,
  "features": [
    { "kind": "Feature", "name": "n_main", "coefficient": 1.0, "exponent": 1.0 }
  ]
}

1 500 MB + 0.0008 MB/point: a 1M-point scatter books ~2 300 MB, a 10M-point one ~9 500 MB. Note the coefficient/intercept split is arbitrary — only their product matters (0.0008 × 1.0 ≡ 1.0 × 0.0008).

Fitted power-law — tercen/fast_tSNE_operator. t-SNE’s footprint grows sub-linearly with observations and weakly with dimensions, so the fit found fractional exponents over two interacting features:

{
  "kind": "OperatorEstimateModel",
  "intercept": 0.0076,
  "offset": 238.5,
  "features": [
    { "kind": "Feature", "name": "n_cols", "coefficient": 1.0, "exponent": 0.9189 },
    { "kind": "Feature", "name": "n_rows", "coefficient": 1.0, "exponent": 0.3136 }
  ]
}

For 100 000 cells × 20 markers: 0.0076 × 100000^0.9189 × 20^0.3136 + 1.5 × 238.5 ≈ 0.0076 × 39 000 × 2.56 + 358 ≈ 1 117 MB. The two features multiply — that is the interaction between how many observations you embed and how many dimensions each carries.

14.3.5 Fit it from real runs, don’t guess

Tercen records ground truth for you: every operator run stores its actual memory peak on the task document — stats_d_actual_ram_peak (total cgroup peak, includes reclaimable file cache) and stats_d_actual_ram_peak_anon (bytes the operator actually allocated; the better fitting target). The task also stores stats_d_qt_nrows, stats_d_col_nrows, stats_d_row_nrows — the feature values. The recipe:

Collect the peaks and dimensions from historical runs (the more varied the sizes, the better the fit).
Look at the scatter first. Flat cloud → constant-dominated (fit the floor, token slope). Straight line → linear. Bends on a log-log plot → power law with fractional exponent.
Fit, then verify the safety property: the model must predict at or above every observed peak, with 1.3–2× headroom on the median. A memory model that under-books reintroduces the exact failure it exists to prevent; over-booking merely schedules conservatively.
Check the extrapolation at 10× your largest observed run — the number should be defensible, because one day someone will run it.

tercen/plot_operator ships an estimate-memory.yml GitHub workflow that automates this fit — a good starting point if you want it continuous.

Tip

Failed (exit-137) runs are censored observations: they tell you the true need exceeded that run’s limit, not what it was. Fit on successful runs, then check the model clears every failed run’s limit at its recorded dimensions.

Because the file is read from git at install/update time, shipping or changing it is a normal release (bump the container pin, tag) — but it touches no code: results, the unit test, and the image content are unchanged.

14.4 Testing and Validation

Comprehensive testing ensures your operator works correctly across different data scenarios and edge cases.

Testing Approaches

Tercen supports two main testing frameworks: 1. Unit Tests: Simple data files with expected input/output and test specifications 2. Integration Tests: Actual Tercen workflows triggered to perform computations

14.4.1 Unit Test Structure

Create a tests directory in your operator repository with the following structure:

tests/
├── input.csv          # Sample input data
├── output.csv         # Expected output data  
├── test.json          # Test configuration

For multiple test scenarios, use numbered files: - test_1.json, test_2.json for different parameter settings - input_1.csv, input_2.csv for different data scenarios

14.4.2 Creating Comprehensive Test Data

Design test cases that cover various scenarios:

Test Scenario	Purpose	Example Data
Normal Case	Standard operation	Regular numeric data with good distribution
Edge Cases	Boundary conditions	Minimum data points, extreme values
Error Cases	Invalid inputs	Missing data, wrong data types

14.4.3 Test Configuration File

Create a tests/test.json in OperatorUnitTest format. This is what the release install-check and the library gate execute; R/testthat scripts do not count for the gate. A complete, battle-tested reference: pamgene/cs_uka_operator/tests/.

{
  "kind": "OperatorUnitTest",
  "name": "default_params",
  "namespace": "ds0",
  "inputDataUri": "test_in.csv",
  "outputDataUri": ["test_out_1.csv", "test_out_2.csv"],
  "columns": ["Supergroup"],
  "rows": ["ID"],
  "colors": [],
  "labels": ["colSeq"],
  "yAxis": "value",
  "xAxis": "grp",
  "equalityMethod": "R2",
  "r2": 0.99,
  "propertyValues": [
    {"kind": "PropertyValue", "name": "UkaDbVersion", "value": "0.6"},
    {"kind": "PropertyValue", "name": "Seed", "value": "42.0"}
  ]
}

Rules that each come from a real gate failure:

Pin every operator property via propertyValues (a list of PropertyValue objects — values as strings). A fully pinned test never breaks when a bundled database or a default changes; it only changes when you change it. (Note: the field is propertyValues, not properties.)
equalityMethod: "R2" for computed floats. The default equals compares doubles near-exactly and fails on floating-point noise (task.test.operator.bad.value). Reserve equals for exact integer/string outputs.
One entry in outputDataUri per output relation — a result joined to the crosstab typically yields 2+ relations; a count mismatch fails with task.test.operator.bad.nRelations.
Pin column types with .schema sidecars: for each expected CSV also commit <name>.csv.schema — the result relation’s TableSchema JSON (columns with name+type, id/rev stripped). Without it the gate re-infers types from CSV text and fails with task.test.operator.bad.column.type — expected double found int32 on columns of whole numbers. Write doubles type-faithfully too (58.0, not 58).

14.4.4 Determinism first: the Seed rule

A golden test is impossible if two runs differ. Any operator using randomness (permutations, sampling, clustering) must seed the RNG, best as a visible setting:

seed <- ctx$op.value("Seed", as.double, 42)
if (seed >= 0) set.seed(seed)

with a matching Seed DoubleProperty (default 42; -1 = unseeded). Acceptance test: run twice, outputs byte-identical.

14.4.5 Generating and regenerating the expected output

The expected CSVs are golden files from a real run of the exact code being released — they cannot be hand-written. Generate them on Tercen Studio (see the appendix), never on a production instance: install the operator from the release branch, import the small input, run the step with the projection and pinned settings from test.json, run it twice (determinism), export each output relation and its schema.

When a bundled database or intended behaviour changes, the release install-check goes red — that is the alarm: re-run the same input with the new settings, replace the expected files, update the changed propertyValues, bump the version, release. Ship this recipe in the operator repo itself as a CLAUDE.md (the create-operator skill provides a template), so any future maintenance session knows the drill.

This completes our comprehensive guide to Tercen operator development. You now have all the tools and knowledge needed to create robust, efficient, and user-friendly operators that extend Tercen’s analytical capabilities!

# Operator Improvements and Best Practices This chapter covers advanced techniques for improving operator quality, reliability, and maintainability. You'll learn essential practices for logging, error handling, testing, and optimization that ensure your operators work robustly in production environments. ::: {.callout-note} ## Learning Objectives By the end of this chapter, you will be able to: - Implement comprehensive logging and debugging strategies - Build robust error handling and input validation - Create comprehensive test suites for operators - Apply performance optimization techniques - Follow best practices for production-ready operators ::: ::: {.callout-note} ## Prerequisites Before proceeding, ensure you've completed: - [Basic Implementation](4-basic-implementation.qmd) chapter for core operator concepts - [Advanced Features](5-advanced-features.qmd) chapter for complex functionality - [Data Input and Output Patterns](10-input-output-patterns.qmd) for data handling ::: ## Logging and Debugging Effective logging is essential for monitoring operator behavior and diagnosing issues in production environments. ### Basic Logging Implement logging for production operators: ::: {.panel-tabset} ### R ```r ctx$log("Your message.") ``` ### Python ```python ctx.log("Your message.") ``` ::: ::: {.callout-tip} ## Logging Best Practices - **Log key milestones**: Start/end of major operations - **Include data metrics**: Row counts, processing times, memory usage - **Log parameter values**: Help reproduce issues with specific inputs - **Use structured formats**: Enable easier log parsing and analysis - **Avoid logging sensitive data**: Protect user privacy and security ::: ## Error Handling Implement comprehensive error handling that provides helpful feedback to users: ::: {.panel-tabset} ### R ```r # Comprehensive error handling with user-friendly messages robust_operator <- function(ctx) { tryCatch({ # Validate inputs first validate_inputs(ctx) # Main processing with progress logging ctx$log(paste("[INFO]", Sys.time(), "- Beginning data analysis")) # Check for edge cases data <- ctx$select(.ri, .ci, .y) if (any(is.infinite(data$.y))) { ctx$log(paste("[WARNING]", Sys.time(), "- Infinite values detected, removing them")) data <- data[is.finite(data$.y), ] } if (nrow(data) == 0) { stop("No valid data remaining after cleaning") } # Perform analysis result <- perform_analysis(data) ctx$log(paste("[INFO]", Sys.time(), "- Analysis completed successfully")) return(result) }, error = function(e) { # Log the technical error ctx$log(paste("[ERROR]", Sys.time(), "- Technical error:", e$message)) # Provide user-friendly error message if (grepl("projection.*required", e$message)) { stop("Please ensure you have dragged the required data columns to the appropriate axes.") } else if (grepl("data points required", e$message)) { stop("This analysis requires at least 3 data points. Please check your data selection.") } else if (grepl("values must vary", e$message)) { stop("The data values do not vary enough for this analysis. Please check your input data.") } else { stop(paste("An error occurred during analysis:", e$message)) } }) } ``` ### Python ```python def robust_operator(tercen_ctx): """Operator with comprehensive error handling""" try: # Validate inputs first validate_inputs(tercen_ctx) # Main processing with progress logging tercen_ctx.log(f"[INFO] {datetime.now()} - Beginning data analysis") # Check for edge cases df = tercen_ctx.select(['.ri', '.ci', '.y'], df_lib="polars") # Handle infinite values infinite_count = df.filter(pl.col('.y').is_infinite()).height if infinite_count > 0: tercen_ctx.log(f"[WARNING] {datetime.now()} - Infinite values detected ({infinite_count}), removing them") df = df.filter(pl.col('.y').is_finite()) if len(df) == 0: raise ValueError("No valid data remaining after cleaning") # Perform analysis result = perform_analysis(df) tercen_ctx.log(f"[INFO] {datetime.now()} - Analysis completed successfully") return result except ValueError as ve: tercen_ctx.log(f"[ERROR] {datetime.now()} - Validation error: {str(ve)}") # Provide user-friendly error messages if "projection" in str(ve) and "required" in str(ve): raise ValueError("Please ensure you have dragged the required data columns to the appropriate axes.") elif "data points required" in str(ve): raise ValueError("This analysis requires at least 3 data points. Please check your data selection.") elif "values must vary" in str(ve): raise ValueError("The data values do not vary enough for this analysis. Please check your input data.") else: raise ValueError(f"An error occurred during analysis: {str(ve)}") except Exception as e: tercen_ctx.log(f"[ERROR] {datetime.now()} - Unexpected error: {str(e)}") raise ValueError(f"An unexpected error occurred. Please check your data and try again. Error: {str(e)}") ``` ::: ::: {.callout-warning} ## Error Handling Guidelines - **Validate early**: Check inputs before expensive computations - **Fail gracefully**: Provide clear, actionable error messages - **Log technical details**: Help with debugging while keeping user messages simple - **Handle edge cases**: Account for missing data, infinite values, empty datasets - **Test error scenarios**: Ensure error handling works as expected ::: ## Memory: tell Tercen what your operator needs When Tercen runs your operator it puts the container under a hard memory limit. **If the operator doesn't declare its needs, the limit defaults to about 500 MiB.** An operator whose real usage sits near or above that will fail *intermittently* with: ``` Failed to run operator, please increase memory resources. (exit code 137) ``` "Intermittently" is the tell: the run sits exactly on its ceiling, and whether the kernel can reclaim enough cache decides its fate. Same data, different outcome. ### The quick fix (no release needed) On the failing workflow step, add the environment setting `ram` with the value in bytes (for example `2000000000` for 2 GB), reset the step and rerun. A user-defined value is always respected verbatim. ### The durable fix: `memory_model.json` Ship a `memory_model.json` at the repository root. Tercen reads it at install/update time and books memory per run based on the projected data: ```json { "kind": "OperatorEstimateModel", "intercept": 1.0, "offset": 500.0, "features": [ { "kind": "Feature", "name": "n_main", "coefficient": 0.02, "exponent": 1.0 } ] } ``` The estimate, in **MB**, is `intercept × Π(coefficient × feature^exponent) + 1.5 × offset` — the example books 750 MB plus 0.02 MB per projected cell. Available features: | Feature | Value | |---|---| | `n_main` | number of projected crosstab cells | | `n_cols` | rows of the column table | | `n_rows` | rows of the row table | | `settings.<Name>` | numeric value of an operator setting | | `a_||_b` | interaction: product of two features | ::: {.callout-warning} Prefer the schema features. A `settings.*` feature whose setting is left blank evaluates to 0, which can silently disable the whole model (an infinite or zero estimate is discarded). ::: ### How the formula is evaluated — worked example The model is **multiplicative in the features, plus a constant**: ``` estimate_MB = intercept × ∏ ( coefficient_i × feature_i ^ exponent_i ) + 1.5 × offset ``` Take the model above and a crosstab of 200 kinases × 12 samples, so `n_main = 2 400` cells: ``` growth = 1.0 × (0.02 × 2400^1.0) = 48 MB constant = 1.5 × 500 = 750 MB booked = 48 + 750 = 798 MB ``` At 100 000 cells the same model books 750 + 2 000 = 2 750 MB — the booking follows the data instead of being a fixed number. Two structural consequences of this shape are worth internalising: - `offset` is the **additive floor** (times 1.5). It represents everything that exists before your data does: the R or Python runtime, loaded libraries, reference databases read into memory. For many operators this constant is the whole story. - `intercept` and the features form the **growth term**. With several features the terms *multiply*, so two linear features model an `n × m` surface, not a sum. If you want additive behaviour, put it in `offset` and keep one feature. ### Three real models, three shapes **Constant-dominated — `pamgene/cs_uka_operator`.** Measured production runs peaked at 525–592 MB whether the crosstab had 600 cells or 12 000: the UKA reference database and the permutation machinery dominate; the projected data is a rounding error. The model is therefore mostly floor, with a token slope as insurance: ```json { "kind": "OperatorEstimateModel", "intercept": 1.0, "offset": 500.0, "features": [ { "kind": "Feature", "name": "n_main", "coefficient": 0.02, "exponent": 1.0 } ] } ``` `750 MB + 0.02 MB/cell` — fitted against 203 recorded runs, covering every one with 1.3–1.9× headroom. (Before this model existed, the operator got the 500 MiB default: every run sat exactly on its ceiling, and the intermittent exit-137 reports rolled in for months.) **Linear in the data — `tercen/plot_operator`.** A plot's memory scales with the points drawn, on top of an R baseline: ```json { "kind": "OperatorEstimateModel", "intercept": 0.0008, "offset": 1000.0, "features": [ { "kind": "Feature", "name": "n_main", "coefficient": 1.0, "exponent": 1.0 } ] } ``` `1 500 MB + 0.0008 MB/point`: a 1M-point scatter books ~2 300 MB, a 10M-point one ~9 500 MB. Note the coefficient/intercept split is arbitrary — only their product matters (`0.0008 × 1.0 ≡ 1.0 × 0.0008`). **Fitted power-law — `tercen/fast_tSNE_operator`.** t-SNE's footprint grows sub-linearly with observations and weakly with dimensions, so the fit found fractional exponents over two interacting features: ```json { "kind": "OperatorEstimateModel", "intercept": 0.0076, "offset": 238.5, "features": [ { "kind": "Feature", "name": "n_cols", "coefficient": 1.0, "exponent": 0.9189 }, { "kind": "Feature", "name": "n_rows", "coefficient": 1.0, "exponent": 0.3136 } ] } ``` For 100 000 cells × 20 markers: `0.0076 × 100000^0.9189 × 20^0.3136 + 1.5 × 238.5 ≈ 0.0076 × 39 000 × 2.56 + 358 ≈ 1 117 MB`. The two features multiply — that is the interaction between how many observations you embed and how many dimensions each carries. ### Fit it from real runs, don't guess Tercen records ground truth for you: every operator run stores its actual memory peak on the task document — `stats_d_actual_ram_peak` (total cgroup peak, includes reclaimable file cache) and `stats_d_actual_ram_peak_anon` (bytes the operator actually allocated; the better fitting target). The task also stores `stats_d_qt_nrows`, `stats_d_col_nrows`, `stats_d_row_nrows` — the feature values. The recipe: 1. **Collect** the peaks and dimensions from historical runs (the more varied the sizes, the better the fit). 2. **Look at the scatter first.** Flat cloud → constant-dominated (fit the floor, token slope). Straight line → linear. Bends on a log-log plot → power law with fractional exponent. 3. **Fit, then verify the safety property**: the model must predict **at or above every observed peak**, with 1.3–2× headroom on the median. A memory model that under-books reintroduces the exact failure it exists to prevent; over-booking merely schedules conservatively. 4. **Check the extrapolation** at 10× your largest observed run — the number should be defensible, because one day someone will run it. `tercen/plot_operator` ships an `estimate-memory.yml` GitHub workflow that automates this fit — a good starting point if you want it continuous. ::: {.callout-tip} Failed (exit-137) runs are *censored* observations: they tell you the true need exceeded that run's limit, not what it was. Fit on successful runs, then check the model clears every failed run's limit at its recorded dimensions. ::: Because the file is read from git at install/update time, shipping or changing it is a normal release (bump the container pin, tag) — but it touches no code: results, the unit test, and the image content are unchanged. ## Testing and Validation Comprehensive testing ensures your operator works correctly across different data scenarios and edge cases. ::: {.callout-note} ## Testing Approaches Tercen supports two main testing frameworks: 1. **Unit Tests**: Simple data files with expected input/output and test specifications 2. **Integration Tests**: Actual Tercen workflows triggered to perform computations ::: ### Unit Test Structure Create a `tests` directory in your operator repository with the following structure: ``` tests/ ├── input.csv # Sample input data ├── output.csv # Expected output data ├── test.json # Test configuration ``` For multiple test scenarios, use numbered files: - `test_1.json`, `test_2.json` for different parameter settings - `input_1.csv`, `input_2.csv` for different data scenarios ### Creating Comprehensive Test Data Design test cases that cover various scenarios: | Test Scenario | Purpose | Example Data | |---------------|---------|--------------| | **Normal Case** | Standard operation | Regular numeric data with good distribution | | **Edge Cases** | Boundary conditions | Minimum data points, extreme values | | **Error Cases** | Invalid inputs | Missing data, wrong data types | ### Test Configuration File Create a `tests/test.json` in `OperatorUnitTest` format. This is what the release install-check and the library gate execute; R/`testthat` scripts do not count for the gate. A complete, battle-tested reference: [`pamgene/cs_uka_operator/tests/`](https://github.com/pamgene/cs_uka_operator/tree/main/tests). ```json { "kind": "OperatorUnitTest", "name": "default_params", "namespace": "ds0", "inputDataUri": "test_in.csv", "outputDataUri": ["test_out_1.csv", "test_out_2.csv"], "columns": ["Supergroup"], "rows": ["ID"], "colors": [], "labels": ["colSeq"], "yAxis": "value", "xAxis": "grp", "equalityMethod": "R2", "r2": 0.99, "propertyValues": [ {"kind": "PropertyValue", "name": "UkaDbVersion", "value": "0.6"}, {"kind": "PropertyValue", "name": "Seed", "value": "42.0"} ] } ``` Rules that each come from a real gate failure: - **Pin every operator property** via `propertyValues` (a list of `PropertyValue` objects — values as strings). A fully pinned test never breaks when a bundled database or a default changes; it only changes when you change it. (Note: the field is `propertyValues`, not `properties`.) - **`equalityMethod: "R2"` for computed floats.** The default `equals` compares doubles near-exactly and fails on floating-point noise (`task.test.operator.bad.value`). Reserve `equals` for exact integer/string outputs. - **One entry in `outputDataUri` per output relation** — a result joined to the crosstab typically yields 2+ relations; a count mismatch fails with `task.test.operator.bad.nRelations`. - **Pin column types with `.schema` sidecars**: for each expected CSV also commit `<name>.csv.schema` — the result relation's `TableSchema` JSON (columns with name+type, `id`/`rev` stripped). Without it the gate re-infers types from CSV text and fails with `task.test.operator.bad.column.type — expected double found int32` on columns of whole numbers. Write doubles type-faithfully too (`58.0`, not `58`). ### Determinism first: the Seed rule A golden test is impossible if two runs differ. **Any operator using randomness** (permutations, sampling, clustering) **must seed the RNG**, best as a visible setting: ```r seed <- ctx$op.value("Seed", as.double, 42) if (seed >= 0) set.seed(seed) ``` with a matching `Seed` `DoubleProperty` (default 42; `-1` = unseeded). Acceptance test: run twice, outputs byte-identical. ### Generating and regenerating the expected output The expected CSVs are **golden files from a real run of the exact code being released** — they cannot be hand-written. Generate them on **Tercen Studio** (see the appendix), never on a production instance: install the operator from the release branch, import the small input, run the step with the projection and pinned settings from `test.json`, run it twice (determinism), export each output relation and its schema. When a bundled database or intended behaviour changes, the release install-check goes **red — that is the alarm**: re-run the same input with the new settings, replace the expected files, update the changed `propertyValues`, bump the version, release. Ship this recipe in the operator repo itself as a `CLAUDE.md` (the `create-operator` skill provides a template), so any future maintenance session knows the drill. This completes our comprehensive guide to Tercen operator development. You now have all the tools and knowledge needed to create robust, efficient, and user-friendly operators that extend Tercen's analytical capabilities!