ScyllaDB Manager 3.0

The ScyllaDB Manager team is pleased to announce the release of Scylla Manager 3.0, a production-ready version of ScyllaDB Manager for ScyllaDB Enterprise customers and ScyllaDB Open Source users. ScyllaDB Manager is a centralized cluster administration and recurrent tasks automation tool.

ScyllaDB Manager 3.0 brings a new Maintenance Window API, a more flexible time/date format and a new Manager CLI tool, sctool, syntax. More below.

ScyllaDB Enterprise customers are encouraged to upgrade to ScyllaDB Manager 3.0 in coordination with the ScyllaDB support team.

The new release includes upgrades of both ScyllaDB Manager Server and Agent.

Useful Links:

Download ScyllaDB Manager and ScyllaDB Manager Agent for ScyllaDB Enterprise customers
Download ScyllaDB Manager and ScyllaDB Manager Agent for open source users (up to 5 nodes)
ScyllaDB Manager Docker instance and example
ScyllaDB Manager 3.0 documentation
Upgrade from ScyllaDB Manager 2.5 to ScyllaDB Manager 3.0
Submit a ticket for questions or issues with Manager 3.0 (ScyllaDB Enterprise users)

OS Support

Deprecated Ubuntu 16.04 and Debian 9. Both were announced as deprecated in ScyllaDB 4.6 and are no longer supported in ScyllaDB 5.0.
Added support for Debian 11, Ubuntu 22

Maintenance Window

ScyllaDB Manager 3.0 brings a new Maintenance Window API, allowing you to define a date/time windows where scheduled tasks should not run. This is useful for planned activities, like upgrades, when you want to avoid cluster operations.

Alongside the --cron flag (below) there is a new --window flag. The window times are also evaluated against the time zone provided by the --timezone flag.

The execution windows are specified as a comma separated list of time markers s_1,e_1,s_2,e_2...,s_n,e_n where s_i and e_i indicate the start and end of the i-th execution window respectively. The time marker is specified as [WEEKDAY-]HH:MM where

WEEKDAY can be written as the whole word or only using the first 3 characters,
HH:MM is an hour from 00:00 to 23:59.

Example

sctool repair --window MON-00:00,FRI-15:00 can be executed from Monday (starting at midnight) to Friday 3PM.
sctool repair --window 23:00,06:00 can be executed every night from 11PM to 6AM.
sctool repair --window 23:00,06:00,SAT-00:00,SUN-23:59 can be executed every night from 11PM to 6AM and all day during the weekend.

New Time/Date Format

The --start-time and --interval parameters are deprecated and a new --cron parameter is added (see above). It supports the extended syntax including @monthly, @weekly, @daily, @midnight, @hourly, @every X[h|m|s]. The cron expression is evaluated against a time zone. By default we take the time zone where sctool runs but you may override it with --timezone flag. You may have two tasks with the same cron expression running at different times because they have different time zones.

You can get the cron expression explained in plain English in

sctool info
sctool backup --dry-run
sctool repair --dry-run

Example:

Sctool Suspend

The sctool suspend is supplemented with two additional flags --duration and --on-resume-start-tasks.

Without the duration flag it works in legacy mode; i.e. calls the suspend REST API and suspends immediately. With --cron or --duration flags set a suspend task is created (you can give it a name). On execution, if the duration flag is set a suspend/resume task is created; this is a special task to track automatic cluster resuming.

New Sctool Syntax

Sctool syntax is simplified, removing the redundant “task” option from the command. This makes the command shorter and easier to read and write. For example, listing all tasks:

Manager 2.6: sctool task list
Manager 3.0: sctool tasks

sctool tasks shows the task listing, below an example:

You can notice here:

All health check tasks have now a common type
Task names instead of IDs
Cron schedule
Window
Time zone for cron and window
Success and Error counters
There is no arguments printed out

There is a --show-ids flag that shows the IDs instead of names at all times.

sctool info command is the old sctool task history supplemented with task arguments.

You can notice here:

There is no need to specify task ID or name if there is a single task of that type, more on that later
There is cron explained in plain english, it also supports intervals in that case you would have Cron: 7d
End time is removed (we have start + duration)
Error cause is added, the length of this field is limited so if the msg is very long it would be cut and appended with ellipsis

Sctool Task Names

You can give tasks names using --name flag on task creation or update. Task names must be unique for the cluster and task type. That means you can have repair/all and backup/all tasks just fine.

This has changed from the original plan where we wanted to have a possibility to use short task IDs and unique names across all clusters.

From the old doc:

sctool repair progress repair_foo

sctool repair progress 8903d78c

is enough to get the task progress.

It’s not practical to have name conflicts with other clusters, and it’s not practical to have name conflicts with other tasks. It forces addition of artificial prefixes and is cumbersome. So instead of naming a repair task repair_foo we name it foo and refer to it as repair/foo.

Task ID or Name is Optional

If there is only one instance in the cluster of the given task type the name is not needed. This allows to implement commands like

sctool info repair
sctool progress repair
sctool repair update --some-param

without asking the user to specify repair ID or name. If there is more than one task sctool would print a list of available options. It would even suggest tasks in case of misspelled task names.

Consider the following examples.

$ sctool info repair
Error: task ambiguity, use one of:
- repair/system_auth
- repair/d530bf4a-948c-4280-b18b-15469fa89c5f
- repair/bar
- repair/all-weekly
- repair/foo

and

sctool info repair/system_aud
Error: not found, did you mean repair/system_auth

Sctool Completion

Like many tools sctool allows to source <(sctool completion bash) to get bash completion in any environment. It also supports zsh and fish.

Task Statuses

Task statuses are: NEW, RUNNING, STOPPED, DONE, ERROR, ABORTED

In this release we added two more:

STOPPING: a temporary status when the stop command was called but the task did not stop yet.
WAITING: indicates the task was stopped due to a window end and is waiting for the new window.

Task Database Migrations and Legacy Tasks

The health check tasks are migrated to conform to the new standard. There is only one task type (“healthcheck”) and they are automatically named.

Tasks with old schedules (start-date and interval) work fine, they are not migrated. You can see that the task uses the legacy schedule because it would show the schedule like “7d” or “1d” (typically repair and backup respectively) in Schedule in sctool tasks output.

Sctool Deprecated Commands

The deprecated commands are not part of help but users can still call them. In the source tree they are moved to the command/legacy package.

DEPRECATED	NEW
`sctool backup files`
`sctool task delete`	`sctool stop --delete`
`sctool task history`	`sctool info`
`sctool task list`	`sctool tasks`
`sctool task progress`	`sctool progress`
`sctool task start`	`sctool start`
`sctool task stop`	`sctool stop`
`sctool task update`	`sctool [backup\|repair\|suspend] update`

Task Retry Exponential Backoff

Failing task are now handled with exponential backoff not const backoff

Each task has

-r flag – Number of retries
--retry-wait duration – Initial exponential backoff duration X[h|m|s] (default “10m“)

Documentation

The online documentation of sctool is completely generated from the command help and shall be always in sync with help.

More about the doc automation work on David Garcia’s blog.

07 Jun 2022