Skip to content

Commit

Permalink
[PG-810] PG-17 Support (#463)
Browse files Browse the repository at this point in the history
* Temporary disable all workflows

* Add build workflow with PG17

* Fix incompatibilities

* Fix 007_settings_pgsm_query_shared_buffer.pl test

* Fix 018_column_names.pl

* Fix 025_compare_pgss.pl

* Remove tuplestore_donestoring usage at all

* Rename I/O timing statistics columns to shared_blk_{read|write}_time

* Fix comments with fileds numbers

* Fix format

* Revert "Temporary disable all workflows"

This reverts commit 12e75be.

* Disable all workflows except check and build for PG 15, 16 and 17

* Fix

* Fix comments

* Fix migration

* Use REL_17_BETA1 in CI

* Add timers tests to 028_temp_block.pl

* Add local blocks timing statistics columns local_blk_{write|read}_time

* Fix t/027_local_blocks.pl test for older PG versions

* Fix

* Add jit_deform_{count|time} metrics

* Fix

* Add stats_since and minmax_stats_since fields

* Revert "Disable all workflows except check and build for PG 15, 16 and 17"

This reverts commit 73febf3.

* Fix t/028_temp_block.pl for PG14 and below

* Fix build for PG12

* Add pgdg workflow for PG17

* Try to fix PG pgdg workflow

* Fixes and formatting

* Format code

* Add level tracking regression test

* Fix nesting level tracking

* Format code

* Add level tracking test expected result for PG13

* Fix for PG12

* Skip level tracking regression test for PG version less than 14

* Fix toplevel calculation for older PG version

* Fix level tracking test results

* Fix nesting level counting for older PG version

* Revert "Fix nesting level counting for older PG version"

This reverts commit 3e91da8.

* Fix level tracking for older PG versions once again

* Set REL_17_BETA2 tag for PG

* Add CI badge for PG17

* Use PG17 for examples in readme
  • Loading branch information
artemgavrilov authored Jul 18, 2024
1 parent c796995 commit dacb41f
Show file tree
Hide file tree
Showing 24 changed files with 2,336 additions and 196 deletions.
147 changes: 147 additions & 0 deletions .github/workflows/postgresql-17-build.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,147 @@
name: postgresql-17-build
on:
pull_request:
push:
branches:
- main
tags:
- '[0-9]+.[0-9]+.[0-9]+*'

jobs:
build:
name: pg-17-build-test
runs-on: ubuntu-22.04
steps:
- name: Clone postgres repository
uses: actions/checkout@v4
with:
repository: 'postgres/postgres'
ref: 'REL_17_BETA2' # TODO: Replace with REL_17_STABLE

- name: Install dependencies
run: |
sudo apt-get update
sudo apt purge postgresql-client-common postgresql-common \
postgresql postgresql*
sudo apt-get install -y libreadline6-dev systemtap-sdt-dev \
zlib1g-dev libssl-dev libpam0g-dev bison flex \
libipc-run-perl -y docbook-xsl docbook-xsl libxml2 libxml2-utils \
libxml2-dev libxslt-dev xsltproc libkrb5-dev libldap2-dev \
libsystemd-dev gettext tcl-dev libperl-dev pkg-config clang-11 \
llvm-11 llvm-11-dev libselinux1-dev python3-dev \
uuid-dev liblz4-dev
sudo rm -rf /var/lib/postgresql /var/log/postgresql /etc/postgresql \
/usr/lib/postgresql /usr/include/postgresql /usr/share/postgresql \
/etc/postgresql
sudo rm -f /usr/bin/pg_config
sudo /usr/bin/perl -MCPAN -e 'install IPC::Run'
sudo /usr/bin/perl -MCPAN -e 'install Text::Trim'
- name: Create pgsql dir
run: mkdir -p /opt/pgsql

- name: Build postgres
run: |
export PATH="/opt/pgsql/bin:$PATH"
./configure '--build=x86_64-linux-gnu' '--prefix=/usr' \
'--includedir=${prefix}/include' '--mandir=${prefix}/share/man' \
'--infodir=${prefix}/share/info' '--sysconfdir=/etc' \
'--localstatedir=/var' '--libdir=${prefix}/lib/x86_64-linux-gnu' \
'--libexecdir=${prefix}/lib/x86_64-linux-gnu' '--with-icu' \
'--with-tcl' '--with-perl' '--with-python' '--with-pam' \
'--with-openssl' '--with-libxml' '--with-libxslt' '--with-ldap' \
'PYTHON=/usr/bin/python3' '--mandir=/usr/share/postgresql/17/man' \
'--docdir=/usr/share/doc/postgresql-doc-17' '--with-pgport=5432' \
'--sysconfdir=/etc/postgresql-common' '--datarootdir=/usr/share' \
'--datadir=/usr/share/postgresql/17' '--with-uuid=e2fs' \
'--bindir=/usr/lib/postgresql/17/bin' '--enable-tap-tests' \
'--libdir=/usr/lib/x86_64-linux-gnu' '--enable-debug' \
'--libexecdir=/usr/lib/postgresql' '--with-gnu-ld' \
'--includedir=/usr/include/postgresql' '--enable-dtrace' \
'--enable-nls' '--enable-thread-safety' '--disable-rpath' \
'--with-system-tzdata=/usr/share/zoneinfo' '--with-llvm' \
'LLVM_CONFIG=/usr/bin/llvm-config-11' 'CLANG=/usr/bin/clang-11' \
'--with-systemd' '--with-selinux' 'MKDIR_P=/bin/mkdir -p' \
'PROVE=/usr/bin/prove' 'TAR=/bin/tar' 'XSLTPROC=xsltproc --nonet' \
'LDFLAGS=-Wl,-Bsymbolic-functions -Wl,-z,relro -Wl,-z,now' \
'build_alias=x86_64-linux-gnu' '--with-gssapi' \
'CPPFLAGS=-Wdate-time -D_FORTIFY_SOURCE=2' \
'CFLAGS=-g -O2 -fstack-protector-strong -Wformat -Werror=format-security -fno-omit-frame-pointer' \
'CXXFLAGS=-g -O2 -fstack-protector-strong -Wformat -Werror=format-security'
make world
sudo make install-world
- name: Start postgresql cluster
run: |
export PATH="/usr/lib/postgresql/17/bin:$PATH"
sudo cp /usr/lib/postgresql/17/bin/pg_config /usr/bin
initdb -D /opt/pgsql/data
pg_ctl -D /opt/pgsql/data -l logfile start
- name: Clone pg_stat_monitor repository
uses: actions/checkout@v4
with:
path: 'src/pg_stat_monitor'

- name: Build pg_stat_monitor
run: |
make USE_PGXS=1
sudo make USE_PGXS=1 install
working-directory: src/pg_stat_monitor

- name: Configure and Restart Server
run: |
export PATH="/usr/lib/postgresql/17/bin:$PATH"
pg_ctl -D /opt/pgsql/data -l logfile stop
echo "shared_preload_libraries = 'pg_stat_monitor'" >> \
/opt/pgsql/data/postgresql.conf
echo "compute_query_id = regress" >> /opt/pgsql/data/postgresql.conf
pg_ctl -D /opt/pgsql/data -l logfile start
working-directory: src/pg_stat_monitor

- name: Start pg_stat_monitor_tests
run: |
make installcheck
working-directory: src/pg_stat_monitor/

- name: Change dir permissions on fail
if: ${{ failure() }}
run: |
sudo chmod -R ugo+rwx t
sudo chmod -R ugo+rwx tmp_check
exit 2 # regenerate error so that we can upload files in next step
working-directory: src/pg_stat_monitor

- name: Upload logs on fail
if: ${{ failure() }}
uses: actions/upload-artifact@v4
with:
name: Regressions diff and postgresql log
path: |
src/pg_stat_monitor/regression.diffs
src/pg_stat_monitor/regression.out
src/pg_stat_monitor/logfile
src/pg_stat_monitor/t/results/
src/pg_stat_monitor/tmp_check/log/
!src/pg_stat_monitor/tmp_check/**/archives/*
!src/pg_stat_monitor/tmp_check/**/backup/*
!src/pg_stat_monitor/tmp_check/**/pgdata/*
!src/pg_stat_monitor/tmp_check/**/archives/
!src/pg_stat_monitor/tmp_check/**/backup/
!src/pg_stat_monitor/tmp_check/**/pgdata/
if-no-files-found: warn
retention-days: 3

- name: Start Server installcheck-world tests
run: make installcheck-world

- name: Report on installcheck-world test suites fail
uses: actions/upload-artifact@v4
if: ${{ failure() }}
with:
name: Regressions output files of failed testsuite, and pg log
path: |
**/regression.diffs
**/regression.out
src/pg_stat_monitor/logfile
retention-days: 3
93 changes: 93 additions & 0 deletions .github/workflows/postgresql-17-pgdg-package.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,93 @@
name: postgresql-17-pgdg-package
on:
pull_request:
push:
branches:
- main
tags:
- '[0-9]+.[0-9]+.[0-9]+*'

jobs:
build:
name: pg-17-pgdg-package-test
runs-on: ubuntu-22.04
steps:
- name: Clone pg_stat_monitor repository
uses: actions/checkout@v4
with:
path: 'src/pg_stat_monitor'

- name: Delete old postgresql files
run: |
sudo apt-get update
sudo apt purge postgresql-client-common postgresql-common \
postgresql postgresql*
sudo apt-get install -y libreadline6-dev systemtap-sdt-dev wget \
zlib1g-dev libssl-dev libpam0g-dev bison flex libipc-run-perl
sudo rm -rf /var/lib/postgresql /var/log/postgresql /etc/postgresql \
/usr/lib/postgresql /usr/include/postgresql /usr/share/postgresql \
/etc/postgresql
sudo rm -f /usr/bin/pg_config
sudo /usr/bin/perl -MCPAN -e 'install IPC::Run'
sudo /usr/bin/perl -MCPAN -e 'install Text::Trim'
- name: Install PG Distribution Postgresql 17
run: |
sudo sh -c 'echo "deb http://apt.postgresql.org/pub/repos/apt \
$(lsb_release -cs)-pgdg main 17" > /etc/apt/sources.list.d/pgdg.list'
sudo wget --quiet -O - \
https://www.postgresql.org/media/keys/ACCC4CF8.asc |
sudo apt-key add -
sudo apt update
sudo apt -y install postgresql-17 postgresql-server-dev-17
- name: Change src owner to postgres
run: |
sudo chmod o+rx ~
sudo chown -R postgres:postgres src
- name: Build pg_stat_monitor
run: |
sudo -u postgres bash -c 'make USE_PGXS=1'
sudo make USE_PGXS=1 install
working-directory: src/pg_stat_monitor

- name: Start pg_stat_monitor_tests
run: |
sudo service postgresql stop
echo "shared_preload_libraries = 'pg_stat_monitor'" |
sudo tee -a /etc/postgresql/17/main/postgresql.conf
sudo service postgresql start
sudo psql -V
export PG_TEST_PORT_DIR=${GITHUB_WORKSPACE}/src/pg_stat_monitor
echo $PG_TEST_PORT_DIR
sudo -E -u postgres bash -c 'make installcheck USE_PGXS=1'
working-directory: src/pg_stat_monitor

- name: Change dir permissions on fail
if: ${{ failure() }}
run: |
sudo chmod -R ugo+rwx t
sudo chmod -R ugo+rwx tmp_check
exit 2 # regenerate error so that we can upload files in next step
working-directory: src/pg_stat_monitor

- name: Upload logs on fail
if: ${{ failure() }}
uses: actions/upload-artifact@v4
with:
name: Regressions diff and postgresql log
path: |
src/pg_stat_monitor/regression.diffs
src/pg_stat_monitor/regression.out
src/pg_stat_monitor/logfile
src/pg_stat_monitor/t/results/
src/pg_stat_monitor/tmp_check/log/
!src/pg_stat_monitor/tmp_check/**/archives/*
!src/pg_stat_monitor/tmp_check/**/backup/*
!src/pg_stat_monitor/tmp_check/**/pgdata/*
!src/pg_stat_monitor/tmp_check/**/archives/
!src/pg_stat_monitor/tmp_check/**/backup/
!src/pg_stat_monitor/tmp_check/**/pgdata/
if-no-files-found: warn
retention-days: 3
4 changes: 2 additions & 2 deletions Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -4,15 +4,15 @@ MODULE_big = pg_stat_monitor
OBJS = hash_query.o guc.o pg_stat_monitor.o $(WIN32RES)

EXTENSION = pg_stat_monitor
DATA = pg_stat_monitor--2.0.sql pg_stat_monitor--1.0--2.0.sql
DATA = pg_stat_monitor--2.0.sql pg_stat_monitor--1.0--2.0.sql pg_stat_monitor--2.0--2.1.sql

PGFILEDESC = "pg_stat_monitor - execution statistics of SQL statements"

LDFLAGS_SL += $(filter -lm, $(LIBS))

TAP_TESTS = 1
REGRESS_OPTS = --temp-config $(top_srcdir)/contrib/pg_stat_monitor/pg_stat_monitor.conf --inputdir=regression
REGRESS = basic version guc pgsm_query_id functions counters relations database error_insert application_name application_name_unique top_query cmd_type error rows tags user
REGRESS = basic version guc pgsm_query_id functions counters relations database error_insert application_name application_name_unique top_query cmd_type error rows tags user level_tracking

# Disabled because these tests require "shared_preload_libraries=pg_stat_statements",
# which typical installcheck users do not have (e.g. buildfarm clients).
Expand Down
21 changes: 11 additions & 10 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,7 @@
[![postgresql-14-pgdg-package](https://github.com/percona/pg_stat_monitor/actions/workflows/postgresql-14-pgdg-package.yml/badge.svg)](https://github.com/percona/pg_stat_monitor/actions/workflows/postgresql-14-pgdg-package.yml)
[![postgresql-15-pgdg-package](https://github.com/percona/pg_stat_monitor/actions/workflows/postgresql-15-pgdg-package.yml/badge.svg)](https://github.com/percona/pg_stat_monitor/actions/workflows/postgresql-15-pgdg-package.yml)
[![postgresql-16-pgdg-package](https://github.com/percona/pg_stat_monitor/actions/workflows/postgresql-16-pgdg-package.yml/badge.svg)](https://github.com/percona/pg_stat_monitor/actions/workflows/postgresql-16-pgdg-package.yml)
[![postgresql-17-pgdg-package](https://github.com/percona/pg_stat_monitor/actions/workflows/postgresql-17-pgdg-package.yml/badge.svg)](https://github.com/percona/pg_stat_monitor/actions/workflows/postgresql-17-pgdg-package.yml)

[![PGXN version](https://badge.fury.io/pg/pg_stat_monitor.svg)](https://badge.fury.io/pg/pg_stat_monitor)
[![Code coverage](https://codecov.io/gh/percona/pg_stat_monitor/branch/main/graph/badge.svg)](https://codecov.io/gh/percona/pg_stat_monitor)
Expand Down Expand Up @@ -51,7 +52,7 @@ To learn about other features, available in `pg_stat_monitor`, see the [Features

`pg_stat_monitor` supports PostgreSQL versions 11 and above. It is compatible with both PostgreSQL provided by PostgreSQL Global Development Group (PGDG) and [Percona Distribution for PostgreSQL](https://www.percona.com/software/postgresql-distribution).

The `RPM` (for RHEL and CentOS) and the `DEB` (for Debian and Ubuntu) packages are available from Percona repositories for PostgreSQL versions [12](https://www.percona.com/downloads/postgresql-distribution-12/LATEST/), [13](https://www.percona.com/downloads/postgresql-distribution-13/LATEST/), [14](https://www.percona.com/downloads/postgresql-distribution-14/LATEST/), [15](https://www.percona.com/downloads/postgresql-distribution-15/LATEST/) and [16](https://www.percona.com/downloads/postgresql-distribution-16/LATEST/).
The `RPM` (for RHEL and CentOS) and the `DEB` (for Debian and Ubuntu) packages are available from Percona repositories for PostgreSQL versions [12](https://www.percona.com/downloads/postgresql-distribution-12/LATEST/), [13](https://www.percona.com/downloads/postgresql-distribution-13/LATEST/), [14](https://www.percona.com/downloads/postgresql-distribution-14/LATEST/), [15](https://www.percona.com/downloads/postgresql-distribution-15/LATEST/), [16](https://www.percona.com/downloads/postgresql-distribution-16/LATEST/) and [17](https://www.percona.com/downloads/postgresql-distribution-17/LATEST/).

The RPM packages are also available in the official PostgreSQL (PGDG) yum repositories.

Expand All @@ -61,8 +62,8 @@ The `pg_stat_monitor` should work on the latest version of both [Percona Distrib

| **Distribution** | **Version** | **Provider** |
| ---------------- | --------------- | ------------ |
|[Percona Distribution for PostgreSQL](https://www.percona.com/software/postgresql-distribution)| [12](https://www.percona.com/downloads/postgresql-distribution-12/LATEST/), [13](https://www.percona.com/downloads/postgresql-distribution-13/LATEST/), [14](https://www.percona.com/downloads/postgresql-distribution-14/LATEST/), [15](https://www.percona.com/downloads/postgresql-distribution-15/LATEST/) and [16](https://www.percona.com/downloads/postgresql-distribution-16/LATEST/)| Percona|
| PostgreSQL | 12, 13, 14, 15 and 16 | PostgreSQL Global Development Group (PGDG) |
|[Percona Distribution for PostgreSQL](https://www.percona.com/software/postgresql-distribution)| [12](https://www.percona.com/downloads/postgresql-distribution-12/LATEST/), [13](https://www.percona.com/downloads/postgresql-distribution-13/LATEST/), [14](https://www.percona.com/downloads/postgresql-distribution-14/LATEST/), [15](https://www.percona.com/downloads/postgresql-distribution-15/LATEST/), [16](https://www.percona.com/downloads/postgresql-distribution-16/LATEST/) and [17](https://www.percona.com/downloads/postgresql-distribution-17/LATEST/)| Percona|
| PostgreSQL | 12, 13, 14, 15, 16 and 17 | PostgreSQL Global Development Group (PGDG) |


### Features
Expand Down Expand Up @@ -121,16 +122,16 @@ To install `pg_stat_monitor` from Percona repositories, you need to use the `per
percona-release setup ppg-XX
```

Replace XX with the desired PostgreSQL version. For example, to install `pg_stat_monitor ` for PostgreSQL 16, specify `ppg-16`.
Replace XX with the desired PostgreSQL version. For example, to install `pg_stat_monitor ` for PostgreSQL 17, specify `ppg-17`.

3. Install `pg_stat_monitor` package
* For Debian and Ubuntu:
``` sh
apt-get install percona-pg-stat-monitor16
apt-get install percona-pg-stat-monitor17
```
* For RHEL and CentOS:
``` sh
yum install percona-pg-stat-monitor16
yum install percona-pg-stat-monitor17
```

#### Installing from PostgreSQL `yum` repositories
Expand All @@ -143,7 +144,7 @@ Install `pg_stat_monitor`:
dnf install -y pg_stat_monitor_<VERSION>
```
Replace the `VERSION` variable with the PostgreSQL version you are using (e.g. specify `pg_stat_monitor_16` for PostgreSQL 16)
Replace the `VERSION` variable with the PostgreSQL version you are using (e.g. specify `pg_stat_monitor_17` for PostgreSQL 17)
#### Installing from PGXN
Expand Down Expand Up @@ -204,7 +205,7 @@ sudo systemctl restart postgresql.service


```sh
sudo systemctl restart postgresql-16
sudo systemctl restart postgresql-17
```

Create the extension using the [CREATE EXTENSION](https://www.postgresql.org/docs/current/sql-createextension.html) command. Using this command requires the privileges of a superuser or a database owner. Connect to `psql` as a superuser for a database and run the following command:
Expand Down Expand Up @@ -280,7 +281,7 @@ To uninstall `pg_stat_monitor`, do the following:

**Important**: If the `shared_preload_libraries` parameter includes other modules, specify them all for the `ALTER SYSTEM SET` command to keep using them.

3. Restart the `postgresql` instance to apply the changes. The following command restarts PostgreSQL 16. Replace the version value with the one you are using.
3. Restart the `postgresql` instance to apply the changes. The following command restarts PostgreSQL 17. Replace the version value with the one you are using.

* On Debian and Ubuntu:

Expand All @@ -292,7 +293,7 @@ To uninstall `pg_stat_monitor`, do the following:


```sh
sudo systemctl restart postgresql-16
sudo systemctl restart postgresql-17
```

### How we work
Expand Down
2 changes: 2 additions & 0 deletions hash_query.c
Original file line number Diff line number Diff line change
Expand Up @@ -303,6 +303,8 @@ hash_entry_alloc(pgsmSharedState * pgsm, pgsmHashKey * key, int encoding)
memset(&entry->counters, 0, sizeof(Counters));
entry->query_text.query_pos = InvalidDsaPointer;
entry->counters.info.parent_query = InvalidDsaPointer;
entry->stats_since = GetCurrentTimestamp();
entry->minmax_stats_since = entry->stats_since;

/* set the appropriate initial usage count */
/* re-initialize the mutex each time ... we assume no one using it */
Expand Down
Loading

0 comments on commit dacb41f

Please sign in to comment.