Release Events

Projects RADIUSS Home

bgunnar5 released 1.13.0b1.

## [1.13.0b1]

Added

  • API documentation for Merlin’s core codebase
  • New merlin database command to interact with new database functionality
    • When running locally, SQLite will be used as the database. Otherwise your current results backend will be used
    • merlin database info: prints some basic information about the database
    • merlin database get: allows you to retrieve and print entries in the database
    • merlin database delete: allows you to delete entries in the database
  • Added db_scripts/ folder containing several new files all pertaining to database interaction
    • data_models: a module that houses dataclasses that define the format of the data that’s stored in Merlin’s database.
    • db_commands: an interface for user commands of merlin database to be processed
    • merlin_db: houses the MerlinDatabase class, used as the main point of contact for interactions with the database
    • entities/: A folder containing modules that define a structured interface for interacting with persisted data.
    • entity_managers/: A folder containing classes responsible for managing high-level database operations across all entities.
  • Added backends/ folder containing a new OOP way to interact with results backend databases
    • results_backend: houses an abstract class ResultsBackend that defines what every supported backend implement in Merlin
    • redis/: A folder containing the RedisBackend class that defines specific interactions with the Redis database
    • sqlite/: A folder containing the SQLiteBackend class that defines specific interactions with the SQLite database
    • backend_factory: houses a factory class MerlinBackendFactory that initializes an appropriate ResultsBackend instance
  • Added monitors/ folder containing a refactored, OOP approach to handling the merlin monitor command
    • celery_monitor: houses the CeleryMonitor class a concrete subclass of TaskServerMonitor for monitoring Celery task servers
    • monitor_factory: houses a factory class MonitorFactory that initializes an appropriate TaskServerMonitor instance
    • monitor: houses the Monitor class, used as the top-level point of interaction for the monitor command
    • task_server_monitor: houses the TaskServerMonitor ABC class, which serves as a common interface for monitoring task servers
  • A new celery task called mark_run_as_complete that is automatically added to the task queue associated with the final step in a workflow
  • Added support for Python 3.12 and 3.13
  • Added additional tests for the merlin run and merlin purge commands
  • Aliased types to represent different types of pytest fixtures
  • New test condition StepFinishedFilesCount to help search for MERLIN_FINISHED files in output workspaces
  • Added “Unit-tests” GitHub action to run the unit test suite
  • Added CeleryTaskManager context manager to the test suite to ensure tasks are safely purged from queues if tests fail
  • Added command-tests, workflow-tests, and integration-tests to the Makefile
  • Added tests and docs for the new merlin config options
  • Python 3.8 now requires orderly-set==5.3.0 to avoid a bug with the deepdiff library
  • New step ‘Reinstall pip to avoid vendored package corruption’ to CI workflow jobs that use pip
  • New GitHub actions to reduce common code in CI
  • COPYRIGHT file for ownership details
  • New check for copyright headers in the Makefile

Changed

  • Updated the merlin monitor command
    • it will now attempt to restart workflows automatically if a workflow is hanging
    • it utilizes an object oriented approach in the backend now
  • Celery’s default settings have been updated to add:
    • interval_max: 300 -> tasks will retry for up to 5 minutes instead of 1 minute like it previously was
    • new broker_transport_options:
      • socket_timeout: 300 -> increases the socket timeout to 5 minutes instead of the default 2 minutes
      • retry_policy: {timeout: 600} -> sets the maximum amount of time that Celery will keep trying to connect to the broker to 10 minutes
    • broker_connection_timeout: 60 -> establishing a connection to the broker will not timeout for an entire minute now instead of the previous 4 seconds
    • new generic backend settings:
      • result_backend_always_retry: True -> backend will now auto-retry on the event of recoverable exceptions
      • result_backend_max_retries: 20 -> maximum number of retries in the event of recoverable exceptions
    • new Redis specific settings:
      • redis_retry_on_timeout: True -> retries read/write operations on TimeoutError to the Redis server
      • redis_socket_connect_timeout: 300 -> 5 minute socket timeout for connections to Redis
      • redis_socket_timeout: 300 -> 5 minute socket timeout for read/write operations to Redis
      • redis_socket_keepalive: True -> socket TCP keepalive to keep connections healthy to the Redis server
  • The merlin config command:
    • Now defaults to the LaunchIT setup
    • No longer required to have configuration named app.yaml
    • New subcommands:
      • create: Creates a new configuration file
      • update-broker: Updates the broker section of the configuration file
      • update-backend: Updates the results_backend section of the configuration file
      • use: Point your active configuration to a new configuration file
  • The merlin server command no longer modifies the ~/.merlin/app.yaml file by default. Instead, it modifies the ./merlin_server/app.yaml file.
  • Dropped support for Python 3.7
  • Ported all distributed tests of the integration test suite to pytest
    • There is now a commands/ directory and a workflows/ directory under the integration suite to house these tests
    • Removed the “Distributed-tests” GitHub action as these tests will now be run under “Integration-tests”
  • Removed e2e-distributed* definitions from the Makefile
  • Modified GitHub CI to use shared testing servers hosted by LaunchIT rather than the jackalope server
  • CI to use new actions
  • Copyright headers in all files
    • These now point to the LICENSE and COPYRIGHT files
    • LICENSE: Legal permissions (e.g., MIT terms)
    • COPYRIGHT: Ownership, institutional metadata
    • Make commands that change version/copyright year have been modified

Fixed

  • Running Merlin locally no longer requires an app.yaml configuration file
  • Removed dead lgtm link
  • Potential security vulnerabilities related to logging

Deprecated

  • The --steps argument of the merlin monitor command is now deprecated and will be removed in Version 1.14.0.

@lucpeterson @ryannova @doutriaux1 @woutdenolf </small>View Comment