We were unable to construct the commit group for this pull request: None of the pull request's commits have been successfully processed as yet.

Scan the stress run's own logs when the post-stress restart fails #121125

Open
groeneai opened 1:18am on September 20, 2026 wants to merge 0 commits into clickhouse/clickhouse master from
ci-stress-scan-logs-on-restart-failure

Pull Request Overview

  • Opened on September 20, 2026
  • Status Open
  • Commit count 0

Total Delta

0 Total Diff Delta

Open Days

Open 0 weekdays

Test Delta

0 Diff Delta in Test Files
Breakdown by Phase

How long has this pull request spent in each phase of its lifecycle?

Data pending calculation for pull request

Author avatar

Scan the stress run's own logs when the post-stress restart fails

Changelog category (leave one):


  • CI Fix or Improvement (changelog entry is not required)

Changelog entry (a user-readable short description of the changes that goes into CHANGELOG.md):

...

Description

In 30 days 199 stress jobs (158 PRs, 22 on true master, 12 flavours) failed their post-stress
restart. stress_runner.sh:356 was start_server 10 600 || { echo ...; exit 1; }; that explicit
exit 1 precedes :368 check_logs_for_critical_errors, so on all 199 the S3_ERROR No such key
thrown
and Lost forever for SharedMergeTree detectors and dmesg -T > /test_output/dmesg.log
never ran. <Fatal> and signal-9 detection is unaffected: stress_job.py:379-393 greps the same
logs. What is lost is those two data-loss detectors and dmesg.log. Both greps read
clickhouse-server*.log (renamed there at :343), so neither needs a server.

It goes in this caller, not start_server: the function is an end-of-run step, and :356 is the
only such site.

One behavioural consequence, up front. dmesg.log is the kernel-side input to this job's OOM
classifier (stress_job.py:372-377), and :486-488 already turns an OOM-attributed failure into
OK. It is inoperative on this path today, so restoring it will make some currently-red stress jobs
read as OK; its grep matches any container process, not only the server. If they should stay red
instead, that is a small classifier change I can make here; your call.

collect_query_and_trace_logs stays out: it drives clickhouse-local through the same unreachable
object storage, so query_log/trace_log/metric_log stay missing. The three earlier
start_server sites (after 7 start attempts: 10 rows / 1 PR) and upgrade_runner.sh are unchanged.

Validated in the real CI image against the real stress_tests.lib: master emits only the
start-failure row and no dmesg.log; the patch adds both detector rows and the artifact, still
reporting the start failure; deleting the call reverts the result exactly. 3.7 s on a 5 GiB log. In
the reported run the scan would have found nothing; the defect is that it runs on none of the 199.

Related: https://github.com/ClickHouse/ClickHouse/pull/119808
Provenance: Stress test (arm_tsan) on PR #110105 @ 0da1b2081
(report).


Workflow [PR]
Sync PR [sync-upstream/pr/121125]
<!-- CI automatic block end :ci_links: -->

Comments Threads Pending Resolution

Resolved Comment Threads

clickhouse-gh[bot] reviewed on September 20, 2026