patroni.dcs package¶
Submodules¶
Module contents¶
Abstract classes for Distributed Configuration Store.
-
class
patroni.dcs.
AbstractDCS
(config: Dict[str, Any], mpp: AbstractMPP)¶ Bases:
abc.ABC
Abstract representation of DCS modules.
Implementations of a concrete DCS class, using appropriate backend client interfaces, must include the following methods and properties.
Functional methods that are critical in their timing, required to complete within
retry_timeout
period in order to prevent the DCS considered inaccessible, each perform construction of complex data objects:_postgresql_cluster_loader()
:method which processes the structure of data stored in the DCS used to build the
Cluster
object with all relevant associated data.
_mpp_cluster_loader()
:Similar to above but specifically representing MPP group and workers information.
_load_cluster()
:main method for calling specific
loader
method to build theCluster
object representing the state and topology of the cluster.
Functional methods that are critical in their timing and must be written with ACID transaction properties in mind:
attempt_to_acquire_leader()
:method used in the leader race to attempt to acquire the leader lock by creating the leader key in the DCS, if it does not exist.
_update_leader()
:method to update
leader
key in DCS. Relies on Compare-And-Set to ensure the Primary lock key is updated. If this fails to update within theretry_timeout
window the Primary will be demoted.
Functional method that relies on Compare-And-Create to ensure only one member creates the relevant key:
initialize()
:method used in the race for cluster initialization which creates the
initialize
key in the DCS.
DCS backend getter and setter methods and properties:
take_leader()
: method to create a new leader key in the DCS.set_ttl()
: method for setting TTL value in DCS.ttl()
: property which returns the current TTL.set_retry_timeout()
: method for settingretry_timeout
in DCS backend._write_leader_optime()
: compatibility method to write WAL LSN to DCS._write_status()
: method to write WAL LSN for slots to the DCS._write_failsafe()
: method to write cluster topology to the DCS, used by failsafe mechanism.touch_member()
: method to update individual member key in the DCS.set_history_value()
: method to set thehistory
key in the DCS.
DCS setter methods using Compare-And-Set which although important are less critical if they fail, attempts can be retried or may result in warning log messages:
set_failover_value()
: method to create and/or update thefailover
key in the DCS.set_config_value()
: method to create and/or update thefailover
key in the DCS.set_sync_state_value()
: method to set the synchronous statesync
key in the DCS.
DCS data and key removal methods:
delete_sync_state()
:likewise, a method to remove synchronous state
sync
key from the DCS.
delete_cluster()
:method which will remove cluster information from the DCS. Used only from patronictl.
_delete_leader()
:method relies on CAS, used by a member that is the current leader, to remove the
leader
key in the DCS.
cancel_initialization()
:method to remove the
initialize
key for the cluster from the DCS.
If either of the sync_state set or delete methods fail, although not critical, this may result in
Synchronous replication key updated by someone else
messages being logged.Care should be taken to consult each abstract method for any additional information and requirements such as expected exceptions that should be raised in certain conditions and the object types for arguments and return from methods and properties.
-
_CONFIG
= 'config'¶
-
_FAILOVER
= 'failover'¶
-
_FAILSAFE
= 'failsafe'¶
-
_HISTORY
= 'history'¶
-
_INITIALIZE
= 'initialize'¶
-
_LEADER
= 'leader'¶
-
_LEADER_OPTIME
= 'optime/leader'¶
-
_MEMBERS
= 'members/'¶
-
_OPTIME
= 'optime'¶
-
_STATUS
= 'status'¶
-
_SYNC
= 'sync'¶
-
__get_postgresql_cluster
(path: Optional[str] = None) → patroni.dcs.Cluster¶ Low level method to load a
Cluster
object from DCS.- Parameters
path – optional client path in DCS backend to load from.
- Returns
a loaded
Cluster
instance.
-
__init__
(config: Dict[str, Any], mpp: AbstractMPP) → None¶ Prepare DCS paths, MPP object, initial values for state information and processing dependencies.
-
_abc_impl
= <_abc._abc_data object>¶
-
abstract
_delete_leader
(leader: patroni.dcs.Leader) → bool¶ Remove leader key from DCS.
This method should remove leader key if current instance is the leader.
- Parameters
leader –
Leader
object with information about the leader.- Returns
True
if successfully committed to DCS.
-
_get_mpp_cluster
() → patroni.dcs.Cluster¶ Load MPP cluster from DCS.
- Returns
A MPP
Cluster
instance for the coordinator with workers clusters in the Cluster.workers dict.
-
abstract
_load_cluster
(path: str, loader: Callable[[Any], Union[patroni.dcs.Cluster, Dict[int, patroni.dcs.Cluster]]]) → Union[patroni.dcs.Cluster, Dict[int, patroni.dcs.Cluster]]¶ Main abstract method that implements the loading of
Cluster
instance.Note
Internally this method should call the loader method that will build
Cluster
object which represents current state and topology of the cluster in DCS. This method supposed to be called only by theget_cluster()
method.- Parameters
path – the path in DCS where to load Cluster(s) from.
loader – one of
_postgresql_cluster_loader()
or_mpp_cluster_loader()
.
- Raise
DCSError
in case of communication problems with DCS. If the current node was running as a primary and exception raised, instance would be demoted.
-
abstract
_mpp_cluster_loader
(path: Any) → Dict[int, patroni.dcs.Cluster]¶ Load and build all PostgreSQL clusters from a single MPP cluster.
-
abstract
_postgresql_cluster_loader
(path: Any) → patroni.dcs.Cluster¶ Load and build the
Cluster
object from DCS, which represents a single PostgreSQL cluster.
-
_set_loop_wait
(loop_wait: int) → None¶ Set new loop_wait value.
- Parameters
loop_wait – value to set.
-
abstract
_update_leader
(leader: patroni.dcs.Leader) → bool¶ Update
leader
key (or session) ttl.Note
You have to use CAS (Compare And Swap) operation in order to update leader key, for example for etcd
prevValue
parameter must be used.If update fails due to DCS not being accessible or because it is not able to process requests (hopefully temporary), the
DCSError
exception should be raised.- Parameters
leader – a reference to a current
leader
object.- Returns
True
ifleader
key (or session) has been updated successfully.
-
abstract
_write_failsafe
(value: str) → bool¶ Write current cluster topology to DCS that will be used by failsafe mechanism (if enabled).
- Parameters
value – failsafe topology serialized in JSON format.
- Returns
True
if successfully committed to DCS.
-
abstract
_write_leader_optime
(last_lsn: str) → bool¶ Write current WAL LSN into
/optime/leader
key in DCS.- Parameters
last_lsn – absolute WAL LSN in bytes.
- Returns
True
if successfully committed to DCS.
-
abstract
_write_status
(value: str) → bool¶ Write current WAL LSN and
confirmed_flush_lsn
of permanent slots into the/status
key in DCS.- Parameters
value – status serialized in JSON format.
- Returns
True
if successfully committed to DCS.
-
abstract
attempt_to_acquire_leader
() → bool¶ Attempt to acquire leader lock.
Note
This method should create
/leader
key with the value_name
.The key must be created atomically. In case the key already exists it should not be overwritten and
False
must be returned.If key creation fails due to DCS not being accessible or because it is not able to process requests (hopefully temporary), the
DCSError
exception should be raised.- Returns
True
if key has been created successfully.
-
abstract
cancel_initialization
() → bool¶ Removes the
initialize
key for a cluster.- Returns
True
if successfully committed to DCS.
-
client_path
(path: str) → str¶ Construct the absolute key name from appropriate parts for the DCS type.
- Parameters
path – The key name within the current Patroni cluster.
- Returns
absolute key name for the current Patroni cluster.
-
property
cluster
¶ Cached DCS cluster information that has not yet expired.
-
property
config_path
¶ Get the client path for
config
.
-
abstract
delete_cluster
() → bool¶ Delete cluster from DCS.
- Returns
True
if successfully committed to DCS.
-
delete_leader
(leader: Optional[patroni.dcs.Leader], last_lsn: Optional[int] = None) → bool¶ Update
optime/leader
and voluntarily remove leader key from DCS.This method should remove leader key if current instance is the leader.
- Parameters
leader –
Leader
object with information about the leader.last_lsn – latest checkpoint location in bytes.
- Returns
boolean result of called abstract
_delete_leader()
.
-
abstract
delete_sync_state
(version: Optional[Any] = None) → bool¶ Delete the synchronous state from DCS.
- Parameters
version – for conditional deletion of the key/object.
- Returns
True
if delete successful.
-
property
failover_path
¶ Get the client path for
failover
.
-
property
failsafe
¶ Stored value of
_last_failsafe
.
-
property
failsafe_path
¶ Get the client path for
failsafe
.
-
get_cluster
() → patroni.dcs.Cluster¶ Retrieve a fresh view of DCS.
Note
Stores copy of time, status and failsafe values for comparison in DCS update decisions. Caching is required to avoid overhead placed upon the REST API.
Returns either a PostgreSQL or MPP implementation of
Cluster
depending on availability.- Returns
-
get_mpp_coordinator
() → Optional[patroni.dcs.Cluster]¶ Load the PostgreSQL cluster for the MPP Coordinator.
Note
This method is only executed on the worker nodes to find the coordinator.
- Returns
Select
Cluster
instance associated with the MPP Coordinator group ID.
-
property
history_path
¶ Get the client path for
history
.
-
abstract
initialize
(create_new: bool = True, sysid: str = '') → bool¶ Race for cluster initialization.
This method should atomically create
initialize
key and returnTrue
, otherwise it should returnFalse
.- Parameters
create_new –
False
if the key should already exist (in the case we are setting the system_id).sysid – PostgreSQL cluster system identifier, if specified, is written to the key.
- Returns
True
if key has been created successfully.
-
property
initialize_path
¶ Get the client path for
initialize
.
-
is_mpp_coordinator
() → bool¶ Cluster
instance has a Coordinator group ID.- Returns
True
if the given node is running as the MPP Coordinator.
-
property
last_seen
¶ The time recorded when the DCS was last reachable.
-
property
leader_optime_path
¶ Get the client path for
optime/leader
(legacy key, superseded bystatus
).
-
property
leader_path
¶ Get the client path for
leader
.
-
property
loop_wait
¶ The recorded value for cluster HA loop wait time in seconds.
-
manual_failover
(leader: Optional[str], candidate: Optional[str], scheduled_at: Optional[datetime.datetime] = None, version: Optional[Any] = None) → bool¶ Prepare dictionary with given values and set
/failover
key in DCS.- Parameters
leader – value to set for
leader
.candidate – value to set for
member
.scheduled_at – value converted to ISO date format for
scheduled_at
.version – for conditional update of the key/object.
- Returns
True
if successfully committed to DCS.
-
property
member_path
¶ Get the client path for
member
representing this node.
-
property
members_path
¶ Get the client path for
members
.
-
property
mpp
¶ Get the effective underlying MPP, if any has been configured.
-
reload_config
(config: Union[Config, Dict[str, Any]]) → None¶ Load and set relevant values from configuration.
Sets
loop_wait
,ttl
andretry_timeout
properties.- Parameters
config – Loaded configuration information object or dictionary of key value pairs.
-
abstract
set_config_value
(value: str, version: Optional[Any] = None) → bool¶ Create or update
/config
key in DCS.- Parameters
value – new value to set in the
config
key.version – for conditional update of the key/object.
- Returns
True
if successfully committed to DCS.
-
abstract
set_failover_value
(value: str, version: Optional[Any] = None) → bool¶ Create or update
/failover
key.- Parameters
value – value to set.
version – for conditional update of the key/object.
- Returns
True
if successfully committed to DCS.
-
abstract
set_history_value
(value: str) → bool¶ Set value for
history
in DCS.- Parameters
value – new value of
history
key/object.- Returns
True
if successfully committed to DCS.
-
abstract
set_sync_state_value
(value: str, version: Optional[Any] = None) → Union[Any, bool]¶ Set synchronous state in DCS.
- Parameters
value – the new value of
/sync
key.version – for conditional update of the key/object.
- Returns
version of the new object or
False
in case of error.
-
property
status_path
¶ Get the client path for
status
.
-
property
sync_path
¶ Get the client path for
sync
.
-
static
sync_state
(leader: Optional[str], sync_standby: Optional[Collection[str]]) → Dict[str, Any]¶ Build
sync_state
dictionary.- Parameters
leader – name of the leader node that manages
/sync
key.sync_standby – collection of currently known synchronous standby node names.
- Returns
dictionary that later could be serialized to JSON or saved directly to DCS.
-
abstract
take_leader
() → bool¶ Establish a new leader in DCS.
Note
This method should create leader key with value of
_name
andttl
ofttl
.Since it could be called only on initial cluster bootstrap it could create this key regardless, overwriting the key if necessary.
- Returns
True
if successfully committed to DCS.
-
abstract
touch_member
(data: Dict[str, Any]) → bool¶ Update member key in DCS.
Note
This method should create or update key with the name with
/members/
+_name
and the value of data in a given DCS.- Parameters
data – information about an instance (including connection strings).
- Returns
True
if successfully committed to DCS.
-
abstract property
ttl
¶ Get current
ttl
value.
-
update_leader
(leader: patroni.dcs.Leader, last_lsn: Optional[int], slots: Optional[Dict[str, int]] = None, failsafe: Optional[Dict[str, str]] = None) → bool¶ Update
leader
key (or session) ttl and optime/leader.- Parameters
leader –
Leader
object with information about the leader.last_lsn – absolute WAL LSN in bytes.
slots – dictionary with permanent slots
confirmed_flush_lsn
.failsafe – if defined dictionary passed to
write_failsafe()
.
- Returns
True
ifleader
key (or session) has been updated successfully.
-
watch
(leader_version: Optional[Any], timeout: float) → bool¶ Sleep if the current node is a leader, otherwise, watch for changes of leader key with a given timeout.
- Parameters
leader_version – version of a leader key.
timeout – timeout in seconds.
- Returns
if
True
this will reschedule the next run of the HA cycle.
-
write_failsafe
(value: Dict[str, str]) → None¶ Write the
/failsafe
key in DCS.- Parameters
value – dictionary value to set, consisting of the
name
andapi_url
of members.
-
write_leader_optime
(last_lsn: int) → None¶ Write value for WAL LSN to
optime/leader
key in DCS.Note
This method abstracts away the required data structure of
write_status()
, so it is not needed in the caller. However, theoptime/leader
is only written inwrite_status()
when the cluster has members with a Patroni version that is old enough to require it (i.e. the old Patroni version doesn’t understand the new format).- Parameters
last_lsn – absolute WAL LSN in bytes.
-
write_status
(value: Dict[str, Any]) → None¶ Write cluster status to DCS if changed.
Note
The DCS key
/status
was introduced in Patroni version 2.1.0. Previous to this the position of last known leader LSN was stored inoptime/leader
. This method has detection for backwards compatibility of members with a version older than this.- Parameters
value – JSON serializable dictionary with current WAL LSN and
confirmed_flush_lsn
of permanent slots.
-
write_sync_state
(leader: Optional[str], sync_standby: Optional[Collection[str]], version: Optional[Any] = None) → Optional[patroni.dcs.SyncState]¶ Write the new synchronous state to DCS.
Calls
sync_state()
to build a dictionary and then calls DCS specificset_sync_state_value()
.- Parameters
leader – name of the leader node that manages
/sync
key.sync_standby – collection of currently known synchronous standby node names.
version – for conditional update of the key/object.
- Returns
the new
SyncState
object orNone
.
-
class
patroni.dcs.
Cluster
(*args: Any, **kwargs: Any)¶ Bases:
patroni.dcs.Cluster
Immutable object (namedtuple) which represents PostgreSQL or MPP cluster.
Note
We are using an old-style attribute declaration here because otherwise it is not possible to override __new__ method. Without it the workers by default gets always the same
dict
object that could be mutated.Consists of the following fields:
- Variables
initialize – shows whether this cluster has initialization key stored in DC or not.
config – global dynamic configuration, reference to ClusterConfig object.
leader –
Leader
object which represents current leader of the cluster.status –
Status
object which represents the /status key.members – list of:class:` Member` objects, all PostgreSQL cluster members including leader
failover – reference to
Failover
object.sync – reference to
SyncState
object, last observed synchronous replication state.history – reference to TimelineHistory object.
failsafe – failsafe topology. Node is allowed to become the leader only if its name is found in this list.
workers – dictionary of workers of the MPP cluster, optional. Each key representing the group and the corresponding value is a
Cluster
instance.
-
property
__permanent_logical_slots
¶ Dictionary of permanent
logical
replication slots.
-
property
__permanent_physical_slots
¶ Dictionary of permanent
physical
replication slots.
-
property
__permanent_slots
¶ Dictionary of permanent replication slots with their known LSN.
-
_get_members_slots
(name: str, role: str) → Dict[str, Dict[str, str]]¶ Get physical replication slots configuration for members that sourcing from this node.
If the
replicatefrom
tag is set on the member - we should not create the replication slot for it on the current primary, because that member would replicate from elsewhere. We still create the slot if thereplicatefrom
destination member is currently not a member of the cluster (fallback to the primary), or ifreplicatefrom
destination member happens to be the current primary.If the
nostream
tag is set on the member - we should not create the replication slot for it on the current primary or any other member even ifreplicatefrom
is set, becausenostream
disables WAL streaming.Will log an error if:
Conflicting slot names between members are found
- Parameters
name – name of this node.
role – role of this node, if this is a
primary
orstandby_leader
return list of members replicating from this node. If not then return a list of members replicating as cascaded replicas from this node.
- Returns
dictionary of physical replication slots that should exist on a given node.
-
_get_permanent_slots
(postgresql: Postgresql, tags: patroni.tags.Tags, role: str) → Dict[str, Any]¶ Get configured permanent replication slots.
Note
Permanent replication slots are only considered if
use_slots
configuration is enabled. A node that is not supposed to become a leader (nofailover) will not have permanent replication slots. Also node with disabled streaming (nostream) and its cascading followers must not have permanent logical slots due to lack of feedback from node to primary, which makes them unsafe to use.In a standby cluster we only support physical replication slots.
The returned dictionary for a non-standby cluster always contains permanent logical replication slots in order to show a warning if they are not supported by PostgreSQL before v11.
- Parameters
postgresql – reference to
Postgresql
object.tags – reference to an object implementing
Tags
interface.role – role of the node –
primary
,standby_leader
orreplica
.
- Returns
dictionary of permanent slot names mapped to attributes.
-
_has_permanent_logical_slots
(postgresql: Postgresql, member: patroni.tags.Tags) → bool¶ Check if the given member node has permanent
logical
replication slots configured.- Parameters
postgresql – reference to a
Postgresql
object.member – reference to an object implementing
Tags
interface for the node that we are checking permanent logical replication slots for.
- Returns
True
if any detected replications slots arelogical
, otherwiseFalse
.
-
_merge_permanent_slots
(slots: Dict[str, Dict[str, str]], permanent_slots: Dict[str, Any], name: str, major_version: int) → List[str]¶ Merge replication slots for members with permanent_slots.
Perform validation of configured permanent slot name, skipping invalid names.
Will update slots in-line based on
type
of slot,physical
orlogical
, and name of node. Type is assumed to bephysical
if there are no attributes stored as the slot value.- Parameters
slots – Slot names with existing attributes if known.
name – name of this node.
permanent_slots – dictionary containing slot name key and slot information values.
major_version – postgresql major version.
- Returns
List of disabled permanent, logical slot names, if postgresql version < 11.
-
static
empty
() → patroni.dcs.Cluster¶ Produce an empty
Cluster
instance.
-
filter_permanent_slots
(postgresql: Postgresql, slots: Dict[str, int]) → Dict[str, int]¶ Filter out all non-permanent slots from provided slots dict.
- Parameters
postgresql – reference to
Postgresql
object.slots – slot names with LSN values.
- Returns
a
dict
object that contains only slots that are known to be permanent.
-
get_clone_member
(exclude_name: str) → Optional[Union[patroni.dcs.Member, patroni.dcs.Leader]]¶ Get member or leader object to use as clone source.
- Parameters
exclude_name – name of a member name to exclude.
- Returns
a randomly selected candidate member from available running members that are configured to as viable sources for cloning (has tag
clonefrom
in configuration). If no member is appropriate the current leader is used.
-
get_member
(member_name: str, fallback_to_leader: bool = True) → Optional[Union[patroni.dcs.Member, patroni.dcs.Leader]]¶
-
get_replication_slots
(postgresql: Postgresql, member: patroni.tags.Tags, *, role: Optional[str] = None, show_error: bool = False) → Dict[str, Dict[str, Any]]¶ Lookup configured slot names in the DCS, report issues found and merge with permanent slots.
Will log an error if:
Any logical slots are disabled, due to version compatibility, and show_error is
True
.
- Parameters
postgresql – reference to
Postgresql
object.member – reference to an object implementing
Tags
interface.role – role of the node, if not set will be taken from postgresql.
show_error – if
True
report error if any disabled logical slots or conflicting slot names are found.
- Returns
final dictionary of slot names, after merging with permanent slots and performing sanity checks.
-
get_slot_name_on_primary
(name: str, tags: patroni.tags.Tags) → Optional[str]¶ Get the name of physical replication slot for this node on the primary.
Note
P <– I <– L
In case of cascading replication we have to check not our physical slot, but slot of the replica that connects us to the primary.
- Parameters
name – name of the member node to check.
tags – reference to an object implementing
Tags
interface.
- Returns
the slot name on the primary that is in use for physical replication on this node.
-
has_member
(member_name: str) → bool¶ Check if the given member name is present in the cluster.
- Parameters
member_name – name to look up in the
members
.- Returns
True
if the member name is found.
-
has_permanent_slots
(postgresql: Postgresql, member: patroni.tags.Tags) → bool¶ Check if our node has permanent replication slots configured.
- Parameters
postgresql – reference to
Postgresql
object.member – reference to an object implementing
Tags
interface for the node that we are checking permanent logical replication slots for.
- Returns
True
if there are permanent replication slots configured, otherwiseFalse
.
-
is_empty
()¶ Validate definition of all attributes of this
Cluster
instance.- Returns
True
if all attributes of the currentCluster
are unpopulated.
-
static
is_logical_slot
(value: Union[Any, Dict[str, Any]]) → bool¶ Check whether provided configuration is for permanent logical replication slot.
- Parameters
value – configuration of the permanent replication slot.
- Returns
True
if value is a logical replication slot, otherwiseFalse
.
-
static
is_physical_slot
(value: Union[Any, Dict[str, Any]]) → bool¶ Check whether provided configuration is for permanent physical replication slot.
- Parameters
value – configuration of the permanent replication slot.
- Returns
True
if value is a physical replication slot, otherwiseFalse
.
-
is_unlocked
() → bool¶ Check if the cluster does not have the leader.
- Returns
True
if a leader name is not defined.
-
property
last_lsn
¶ Last known leader LSN.
-
property
leader_name
¶ The name of the leader if defined otherwise
None
.
-
property
min_version
¶ Lowest Patroni software version found in known members of the cluster.
-
should_enforce_hot_standby_feedback
(postgresql: Postgresql, member: patroni.tags.Tags) → bool¶ Determine whether
hot_standby_feedback
should be enabled for the given member.The
hot_standby_feedback
must be enabled if the current replica haslogical
slots, or it is working as a cascading replica for the other node that haslogical
slots.- Parameters
postgresql – reference to a
Postgresql
object.member – reference to an object implementing
Tags
interface for the node that we are checking permanent logical replication slots for.
- Returns
True
if this node or any member replicating from this node has permanent logical slots, otherwiseFalse
.
-
property
slots
¶ {"slot_name": int}
.- Type
State of permanent replication slots on the primary in the format
-
property
timeline
¶ Get the cluster history index from the
history
.- Returns
If the recorded history is empty assume timeline is
1
, if it is not defined or the stored history is not formatted as expected0
is returned and an error will be logged. Otherwise, the last number stored incremented by 1 is returned.- Example
No history provided: >>> Cluster(0, 0, 0, Status.empty(), 0, 0, 0, 0, None, {}).timeline 0
Empty history assume timeline is
1
: >>> Cluster(0, 0, 0, Status.empty(), 0, 0, 0, TimelineHistory.from_node(1, ‘[]’), None, {}).timeline 1Invalid history format, a string of
a
, returns0
: >>> Cluster(0, 0, 0, Status.empty(), 0, 0, 0, TimelineHistory.from_node(1, ‘[[“a”]]’), None, {}).timeline 0History as a list of strings: >>> history = TimelineHistory.from_node(1, ‘[[“3”, “2”, “1”]]’) >>> Cluster(0, 0, 0, Status.empty(), 0, 0, 0, history, None, {}).timeline 4
-
class
patroni.dcs.
ClusterConfig
(version: Union[int, str], data: Dict[str, Any], modify_version: Union[int, str])¶ Bases:
NamedTuple
Immutable object (namedtuple) which represents cluster configuration.
- Variables
version – version number for the object.
data – dictionary of configuration information.
modify_version – modified version number.
-
_asdict
()¶ Return a new dict which maps field names to their values.
-
_field_defaults
= {}¶
-
_fields
= ('version', 'data', 'modify_version')¶
-
classmethod
_make
(iterable)¶ Make a new ClusterConfig object from a sequence or iterable
-
_replace
(**kwds)¶ Return a new ClusterConfig object replacing specified fields with new values
-
static
from_node
(version: Union[int, str], value: str, modify_version: Optional[Union[int, str]] = None) → patroni.dcs.ClusterConfig¶ Factory method to parse value as configuration information.
- Parameters
version – version number for object.
value – raw JSON serialized data, if not parsable replaced with an empty dictionary.
modify_version – optional modify version number, use version if not provided.
- Returns
constructed
ClusterConfig
instance.- Example
>>> ClusterConfig.from_node(1, '{') is None False
-
class
patroni.dcs.
Failover
(version: Union[int, str], leader: Optional[str], candidate: Optional[str], scheduled_at: Optional[datetime.datetime])¶ Bases:
NamedTuple
Immutable object (namedtuple) representing configuration information required for failover/switchover capability.
- Variables
version – version of the object.
leader – name of the leader. If value isn’t empty we treat it as a switchover from the specified node.
candidate – the name of the member node to be considered as a failover candidate.
scheduled_at – in the case of a switchover the
datetime
object to perform the scheduled switchover.
- Example
>>> 'Failover' in str(Failover.from_node(1, '{"leader": "cluster_leader"}')) True
>>> 'Failover' in str(Failover.from_node(1, {"leader": "cluster_leader"})) True
>>> 'Failover' in str(Failover.from_node(1, '{"leader": "cluster_leader", "member": "cluster_candidate"}')) True
>>> Failover.from_node(1, 'null') is None False
>>> n = '''{"leader": "cluster_leader", "member": "cluster_candidate", ... "scheduled_at": "2016-01-14T10:09:57.1394Z"}'''
>>> 'tzinfo=' in str(Failover.from_node(1, n)) True
>>> Failover.from_node(1, None) is None False
>>> Failover.from_node(1, '{}') is None False
>>> 'abc' in Failover.from_node(1, 'abc:def') True
-
_asdict
()¶ Return a new dict which maps field names to their values.
-
_field_defaults
= {}¶
-
_fields
= ('version', 'leader', 'candidate', 'scheduled_at')¶
-
classmethod
_make
(iterable)¶ Make a new Failover object from a sequence or iterable
-
_replace
(**kwds)¶ Return a new Failover object replacing specified fields with new values
-
static
from_node
(version: Union[int, str], value: Union[str, Dict[str, str]]) → patroni.dcs.Failover¶ Factory method to parse value as failover configuration.
- Parameters
version – version number for the object.
value – JSON serialized data or a dictionary of configuration. Can also be a colon
:
delimited list of leader, followed by candidate name (legacy format). Ifscheduled_at
key is defined the value will be parsed bydateutil.parser.parse()
.
- Returns
constructed
Failover
information object
-
scheduled_at
: Optional[datetime.datetime]¶ Alias for field number 3
-
class
patroni.dcs.
Leader
(version: Union[int, str], session: Optional[Union[int, float, str]], member: patroni.dcs.Member)¶ Bases:
NamedTuple
Immutable object (namedtuple) which represents leader key.
Consists of the following fields:
- Variables
-
_asdict
()¶ Return a new dict which maps field names to their values.
-
_field_defaults
= {}¶
-
_fields
= ('version', 'session', 'member')¶
-
classmethod
_make
(iterable)¶ Make a new Leader object from a sequence or iterable
-
_replace
(**kwds)¶ Return a new Leader object replacing specified fields with new values
-
property
checkpoint_after_promote
¶ Determine whether a checkpoint has occurred for this leader after promotion.
- Returns
True
if the role ismaster
orprimary
andcheckpoint_after_promote
is not set,False
if not amaster
orprimary
or if the checkpoint hasn’t occurred. If the version of Patroni is older than 1.5.6, returnNone
.- Example
>>> Leader(1, '', Member.from_node(1, '', '', '{"version":"z"}')).checkpoint_after_promote
-
conn_kwargs
(auth: Optional[Dict[str, str]] = None) → Dict[str, str]¶ Connection keyword arguments.
- Parameters
auth – an optional dictionary containing authentication information.
- Returns
the result of the called
Member.conn_kwargs()
method.
-
member
: patroni.dcs.Member¶ Alias for field number 2
-
property
name
¶ The leader “member” name.
-
property
timeline
¶ Timeline value of
data
.
-
class
patroni.dcs.
Member
(version: Union[int, str], name: str, session: Optional[Union[int, float, str]], data: Dict[str, Any])¶ Bases:
patroni.tags.Tags
,patroni.dcs.Member
Immutable object (namedtuple) which represents single member of PostgreSQL cluster.
Note
We are using an old-style attribute declaration here because otherwise it is not possible to override
__new__
method in theRemoteMember
class.Note
These two keys in data are always written to the DCS, but care is taken to maintain consistency and resilience from data that is read:
conn_url
: connection string containing host, user and password which could be used to access this member.api_url
: REST API url of patroni instanceConsists of the following fields:
- Variables
-
_abc_impl
= <_abc._abc_data object>¶
-
property
api_url
¶ The
api_url
value fromdata
if defined.
-
property
clonefrom
¶ True
if bothclonefrom
tag isTrue
and a connection URL is defined.
-
conn_kwargs
(auth: Optional[Union[Any, Dict[str, Any]]] = None) → Dict[str, Any]¶ Give keyword arguments used for PostgreSQL connection settings.
- Parameters
auth – Authentication properties - can be defined as anything supported by the
psycopg2
orpsycopg
modules. Converts a key ofusername
touser
if supplied.- Returns
A dictionary containing a merge of default parameter keys
host
,port
anddbname
, with the contents ofdata
conn_kwargs
key. If those are not defined will parse and reform connection parameters fromconn_url
. One of these two attributes needs to have data defined to construct the output dictionary. Finally, auth parameters are merged with the dictionary before returned.
-
property
conn_url
¶ The
conn_url
value fromdata
if defined or constructed fromconn_kwargs
.
-
static
from_node
(version: Union[int, str], name: str, session: Optional[Union[int, float, str]], value: str) → patroni.dcs.Member¶ Factory method for instantiating
Member
from a JSON serialised string or object.- Parameters
version – modification version of a given member key in a Configuration Store.
name – name of PostgreSQL cluster member.
session – either session id or just ttl in seconds.
value – JSON encoded string containing arbitrary data i.e.
conn_url
,api_url
,xlog_location
,state
,role
,tags
, etc. OR a connection URL starting withpostgres://
.
- Returns
an
Member
instance built with the given arguments.- Example
>>> Member.from_node(-1, '', '', '{"conn_url": "postgres://foo@bar/postgres"}') is not None True
>>> Member.from_node(-1, '', '', '{') Member(version=-1, name='', session='', data={})
-
property
lsn
¶ Current LSN (receive/flush/replay).
-
property
patroni_version
¶ The
version
string value fromdata
converted to tuple.- Example
>>> Member.from_node(1, '', '', '{"version":"1.2.3"}').patroni_version (1, 2, 3)
-
property
state
¶ The
state
value ofdata
.
The
tags
value fromdata
if defined, otherwise an empty dictionary.
-
class
patroni.dcs.
RemoteMember
(name: str, data: Dict[str, Any])¶ Bases:
patroni.dcs.Member
Represents a remote member (typically a primary) for a standby cluster.
- Variables
ALLOWED_KEYS – Controls access to relevant key names that could be in stored
data
.
-
ALLOWED_KEYS
: Tuple[str, …] = ('primary_slot_name', 'create_replica_methods', 'restore_command', 'archive_cleanup_command', 'recovery_min_apply_delay', 'no_replication_slot')¶
-
_abc_impl
= <_abc._abc_data object>¶
-
exception
patroni.dcs.
ReturnFalseException
¶ Bases:
Exception
Exception to be caught by the
catch_return_false_exception()
decorator.
-
class
patroni.dcs.
Status
(last_lsn: int, slots: Optional[Dict[str, int]])¶ Bases:
NamedTuple
Immutable object (namedtuple) which represents /status key.
Consists of the following fields:
- Variables
-
_asdict
()¶ Return a new dict which maps field names to their values.
-
_field_defaults
= {}¶
-
_fields
= ('last_lsn', 'slots')¶
-
classmethod
_make
(iterable)¶ Make a new Status object from a sequence or iterable
-
_replace
(**kwds)¶ Return a new Status object replacing specified fields with new values
-
static
empty
() → patroni.dcs.Status¶ Construct an empty
Status
instance.- Returns
empty
Status
object.
-
class
patroni.dcs.
SyncState
(version: Optional[Union[int, str]], leader: Optional[str], sync_standby: Optional[str])¶ Bases:
NamedTuple
Immutable object (namedtuple) which represents last observed synchronous replication state.
- Variables
version – modification version of a synchronization key in a Configuration Store.
leader – reference to member that was leader.
sync_standby – synchronous standby list (comma delimited) which are last synchronized to leader.
-
_asdict
()¶ Return a new dict which maps field names to their values.
-
_field_defaults
= {}¶
-
_fields
= ('version', 'leader', 'sync_standby')¶
-
classmethod
_make
(iterable)¶ Make a new SyncState object from a sequence or iterable
-
_replace
(**kwds)¶ Return a new SyncState object replacing specified fields with new values
-
static
_str_to_list
(value: str) → List[str]¶ Splits a string by comma and returns list of strings.
- Parameters
value – a comma separated string.
- Returns
list of non-empty strings after splitting an input value by comma.
-
static
empty
(version: Optional[Union[int, str]] = None) → patroni.dcs.SyncState¶ Construct an empty
SyncState
instance.- Parameters
version – optional version number.
- Returns
empty synchronisation state object.
-
static
from_node
(version: Optional[Union[int, str]], value: Optional[Union[str, Dict[str, Any]]]) → patroni.dcs.SyncState¶ Factory method to parse value as synchronisation state information.
- Parameters
version – optional version number for the object.
value – (optionally JSON serialised) synchronisation state information
- Returns
constructed
SyncState
object.- Example
>>> SyncState.from_node(1, None).leader is None True
>>> SyncState.from_node(1, '{}').leader is None True
>>> SyncState.from_node(1, '{').leader is None True
>>> SyncState.from_node(1, '[]').leader is None True
>>> SyncState.from_node(1, '{"leader": "leader"}').leader == "leader" True
>>> SyncState.from_node(1, {"leader": "leader"}).leader == "leader" True
-
property
is_empty
¶ True
if/sync
key is not valid (doesn’t have a leader).
-
leader_matches
(name: Optional[str]) → bool¶ Compare the given name to stored leader value.
- Returns
True
if name is matching theleader
value.
-
matches
(name: Optional[str], check_leader: bool = False) → bool¶ Checks if node is presented in the /sync state.
Since PostgreSQL does case-insensitive checks for synchronous_standby_name we do it also.
- Parameters
name – name of the node.
check_leader – by default the name is searched for only in members, a value of
True
will include the leader to list.
- Returns
True
if the/sync
key notis_empty()
and the given name is among those presented in the sync state.- Example
>>> s = SyncState(1, 'foo', 'bar,zoo')
>>> s.matches('foo') False
>>> s.matches('fOo', True) True
>>> s.matches('Bar') True
>>> s.matches('zoO') True
>>> s.matches('baz') False
>>> s.matches(None) False
>>> SyncState.empty(1).matches('foo') False
-
property
members
¶ sync_standby
as list or an empty list if undefined or object consideredempty
.
-
class
patroni.dcs.
TimelineHistory
(version: Union[int, str], value: Any, lines: List[Union[Tuple[int, int, str], Tuple[int, int, str, str], Tuple[int, int, str, str, str]]])¶ Bases:
NamedTuple
Object representing timeline history file.
Note
The content held in lines deserialized from value are lines parsed from PostgreSQL timeline history files, consisting of the timeline number, the LSN where the timeline split and any other string held in the file. The files are parsed by
parse_history()
.- Variables
-
_asdict
()¶ Return a new dict which maps field names to their values.
-
_field_defaults
= {}¶
-
_fields
= ('version', 'value', 'lines')¶
-
classmethod
_make
(iterable)¶ Make a new TimelineHistory object from a sequence or iterable
-
_replace
(**kwds)¶ Return a new TimelineHistory object replacing specified fields with new values
-
static
from_node
(version: Union[int, str], value: str) → patroni.dcs.TimelineHistory¶ Parse the given JSON serialized string as a list of timeline history lines.
- Parameters
version – version number
value – JSON serialized string, consisting of parsed lines of PostgreSQL timeline history files, see
TimelineHistory
.
- Returns
composed timeline history object using parsed lines.
- Example
If the passed value argument is not parsed an empty list of lines is returned:
>>> h = TimelineHistory.from_node(1, 2)
>>> h.lines []
-
lines
: List[Union[Tuple[int, int, str], Tuple[int, int, str, str], Tuple[int, int, str, str, str]]]¶ Alias for field number 2
-
value
: Any¶ Alias for field number 1
-
patroni.dcs.
catch_return_false_exception
(func: Callable[[…], Any]) → Any¶ Decorator function for catching functions raising
ReturnFalseException
.- Parameters
func – function to be wrapped.
- Returns
wrapped function.
-
patroni.dcs.
dcs_modules
() → List[str]¶ Get names of DCS modules, depending on execution environment.
- Returns
list of known module names with absolute python module path namespace, e.g.
patroni.dcs.etcd
.
-
patroni.dcs.
get_dcs
(config: Union[Config, Dict[str, Any]]) → AbstractDCS¶ Attempt to load a Distributed Configuration Store from known available implementations.
Note
Using the list of available DCS classes returned by
iter_classes()
attempt to dynamically instantiate the class that implements a DCS using the abstract classAbstractDCS
.Basic top-level configuration parameters retrieved from config are propagated to the DCS specific config before being passed to the module DCS class.
If no module is found to satisfy configuration then report and log an error. This will cause Patroni to exit.
:raises
PatroniFatalException
: if a load of all available DCS modules have been tried and none succeeded.- Parameters
config – object or dictionary with Patroni configuration. This is normally a representation of the main Patroni
- Returns
The first successfully loaded DCS module which is an implementation of
AbstractDCS
.
-
patroni.dcs.
iter_dcs_classes
(config: Optional[Union[Config, Dict[str, Any]]] = None) → Iterator[Tuple[str, Type[AbstractDCS]]]¶ Attempt to import DCS modules that are present in the given configuration.
Note
If a module successfully imports we can assume that all its requirements are installed.
- Parameters
config – configuration information with possible DCS names as keys. If given, only attempt to import DCS modules defined in the configuration. Else, if
None
, attempt to import any supported DCS module.- Returns
an iterator of tuples, each containing the module
name
and the imported DCS class object.
-
patroni.dcs.
parse_connection_string
(value: str) → Tuple[str, Optional[str]]¶ Split and rejoin a URL string into a connection URL and an API URL.
Note
Original Governor stores connection strings for each cluster members in a following format:
postgres://{username}:{password}@{connect_address}/postgres
Since each of our patroni instances provides their own REST API endpoint, it’s good to store this information in DCS along with PostgreSQL connection string. In order to not introduce new keys and be compatible with original Governor we decided to extend original connection string in a following way:
postgres://{username}:{password}@{connect_address}/postgres?application_name={api_url}
This way original Governor could use such connection string as it is, because of feature of
libpq
library.- Parameters
value – The URL string to split.
- Returns
the connection string stored in DCS split into two parts,
conn_url
andapi_url
.
-
patroni.dcs.
slot_name_from_member_name
(member_name: str) → str¶ Translate member name to valid PostgreSQL slot name.
Note
PostgreSQL’s replication slot names must be valid PostgreSQL names. This function maps the wider space of member names to valid PostgreSQL names. Names have their case lowered, dashes and periods common in hostnames are replaced with underscores, other characters are encoded as their unicode codepoint. Name is truncated to 64 characters. Multiple different member names may map to a single slot name.
- Parameters
member_name – The string to convert to a slot name.
- Returns
The string converted using the rules described above.