RADIUSS

bgunnar5 released 1.13.0b1.

## [1.13.0b1]

Added

API documentation for Merlin’s core codebase
New merlin database command to interact with new database functionality
- When running locally, SQLite will be used as the database. Otherwise your current results backend will be used
- merlin database info: prints some basic information about the database
- merlin database get: allows you to retrieve and print entries in the database
- merlin database delete: allows you to delete entries in the database
Added db_scripts/ folder containing several new files all pertaining to database interaction
- data_models: a module that houses dataclasses that define the format of the data that’s stored in Merlin’s database.
- db_commands: an interface for user commands of merlin database to be processed
- merlin_db: houses the MerlinDatabase class, used as the main point of contact for interactions with the database
- entities/: A folder containing modules that define a structured interface for interacting with persisted data.
- entity_managers/: A folder containing classes responsible for managing high-level database operations across all entities.
Added backends/ folder containing a new OOP way to interact with results backend databases
- results_backend: houses an abstract class ResultsBackend that defines what every supported backend implement in Merlin
- redis/: A folder containing the RedisBackend class that defines specific interactions with the Redis database
- sqlite/: A folder containing the SQLiteBackend class that defines specific interactions with the SQLite database
- backend_factory: houses a factory class MerlinBackendFactory that initializes an appropriate ResultsBackend instance
Added monitors/ folder containing a refactored, OOP approach to handling the merlin monitor command
- celery_monitor: houses the CeleryMonitor class a concrete subclass of TaskServerMonitor for monitoring Celery task servers
- monitor_factory: houses a factory class MonitorFactory that initializes an appropriate TaskServerMonitor instance
- monitor: houses the Monitor class, used as the top-level point of interaction for the monitor command
- task_server_monitor: houses the TaskServerMonitor ABC class, which serves as a common interface for monitoring task servers
A new celery task called mark_run_as_complete that is automatically added to the task queue associated with the final step in a workflow
Added support for Python 3.12 and 3.13
Added additional tests for the merlin run and merlin purge commands
Aliased types to represent different types of pytest fixtures
New test condition StepFinishedFilesCount to help search for MERLIN_FINISHED files in output workspaces
Added “Unit-tests” GitHub action to run the unit test suite
Added CeleryTaskManager context manager to the test suite to ensure tasks are safely purged from queues if tests fail
Added command-tests, workflow-tests, and integration-tests to the Makefile
Added tests and docs for the new merlin config options
Python 3.8 now requires orderly-set==5.3.0 to avoid a bug with the deepdiff library
New step ‘Reinstall pip to avoid vendored package corruption’ to CI workflow jobs that use pip
New GitHub actions to reduce common code in CI
COPYRIGHT file for ownership details
New check for copyright headers in the Makefile

Changed

Updated the merlin monitor command
- it will now attempt to restart workflows automatically if a workflow is hanging
- it utilizes an object oriented approach in the backend now
Celery’s default settings have been updated to add:
- interval_max: 300 -> tasks will retry for up to 5 minutes instead of 1 minute like it previously was
- new broker_transport_options:
  - socket_timeout: 300 -> increases the socket timeout to 5 minutes instead of the default 2 minutes
  - retry_policy: {timeout: 600} -> sets the maximum amount of time that Celery will keep trying to connect to the broker to 10 minutes
- broker_connection_timeout: 60 -> establishing a connection to the broker will not timeout for an entire minute now instead of the previous 4 seconds
- new generic backend settings:
  - result_backend_always_retry: True -> backend will now auto-retry on the event of recoverable exceptions
  - result_backend_max_retries: 20 -> maximum number of retries in the event of recoverable exceptions
- new Redis specific settings:
  - redis_retry_on_timeout: True -> retries read/write operations on TimeoutError to the Redis server
  - redis_socket_connect_timeout: 300 -> 5 minute socket timeout for connections to Redis
  - redis_socket_timeout: 300 -> 5 minute socket timeout for read/write operations to Redis
  - redis_socket_keepalive: True -> socket TCP keepalive to keep connections healthy to the Redis server
The merlin config command:
- Now defaults to the LaunchIT setup
- No longer required to have configuration named app.yaml
- New subcommands:
  - create: Creates a new configuration file
  - update-broker: Updates the broker section of the configuration file
  - update-backend: Updates the results_backend section of the configuration file
  - use: Point your active configuration to a new configuration file
The merlin server command no longer modifies the ~/.merlin/app.yaml file by default. Instead, it modifies the ./merlin_server/app.yaml file.
Dropped support for Python 3.7
Ported all distributed tests of the integration test suite to pytest
- There is now a commands/ directory and a workflows/ directory under the integration suite to house these tests
- Removed the “Distributed-tests” GitHub action as these tests will now be run under “Integration-tests”
Removed e2e-distributed* definitions from the Makefile
Modified GitHub CI to use shared testing servers hosted by LaunchIT rather than the jackalope server
CI to use new actions
Copyright headers in all files
- These now point to the LICENSE and COPYRIGHT files
- LICENSE: Legal permissions (e.g., MIT terms)
- COPYRIGHT: Ownership, institutional metadata
- Make commands that change version/copyright year have been modified

Fixed

Running Merlin locally no longer requires an app.yaml configuration file
Removed dead lgtm link
Potential security vulnerabilities related to logging

Deprecated

The --steps argument of the merlin monitor command is now deprecated and will be removed in Version 1.14.0.

@lucpeterson @ryannova @doutriaux1 @woutdenolf </small>View Comment