Skip to content

Commit

Permalink
fix: remove options in code blocks that were messing with styles + fi…
Browse files Browse the repository at this point in the history
…x cards h3 dark mode
  • Loading branch information
juliamrch committed Aug 12, 2024
1 parent cda47ec commit 9041093
Show file tree
Hide file tree
Showing 2 changed files with 9 additions and 4 deletions.
6 changes: 5 additions & 1 deletion assets/css/custom.css
Original file line number Diff line number Diff line change
Expand Up @@ -91,7 +91,8 @@ color: rgb(82 82 82 / .7)
text-decoration: none;
}

html[class~="dark"] .btn-primary {
html[class~="dark"] .btn-primary
{
color: white;
}

Expand Down Expand Up @@ -351,4 +352,7 @@ html[class~="dark"] .content table:not(.code-block table) th {
text-decoration: none;
color: black;

}
html[class~="dark"] .hextra-feature-card h3 {
color: white;
}
7 changes: 4 additions & 3 deletions content/postmortem/2024-08-02.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,7 @@ description: 'A minor release resulted in a cascade of errors that perturbed pla
date: 2024-08-08T15:31:41+02:00
draft: false
excludeSearch: true
type: docs
---
{{< hextra/hero-subtitle >}}
A minor release resulted in a cascade of errors that perturbed platform stability and customer service. How it went wrong,what we’ve learnt
Expand Down Expand Up @@ -541,22 +542,22 @@ Pulsar scalability is performed through the concept of bundles which are kind of

This new *extensible unload strategy* relies on non-persistent real time replication among brokers. Which means when a first broker is upgraded, it initializes the system topic to synchronize other brokers with load metadata. Unfortunately, the extension failed to initialize properly, and doesn’t create the system topic, but without crashing the broker which will be seen up by our monitoring :

```shell{linenos=table,linenostart=1}
```shell
2024-08-02T08:48:53,344+0000 [CompletableFutureDelayScheduler] WARN org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl - clevercloud-pulsar-broker-c3-n1:8080 failed to wait for owner for serviceUnit:orga_79bb9459-98d7-48bb-91ea-cd64204d8140/logs/0x80000000_0xc0000000; Trying to return the current owner:Optional[clevercloud-pulsar-broker-c3-n1:8443] java.util.concurrent TimeoutException: null
2024-08-02T08:48:53,344+0000 [CompletableFutureDelayScheduler] ERROR org.apache.pulsar.broker.loadbalance.extensions.channel.ServiceUnitStateChannelImpl - clevercloud-pulsar-broker-c3-n1:8080 failed to get active owner broker. serviceUnit:orga_79bb9459-98d7-48bb-91ea-cd64204d8140/logs/0x80000000_0xc0000000, state:Owned, owner:Optional[clevercloud-pulsar-broker-c3-n1:8443]
java.util.concurrent.CompletionException: java.lang.IllegalStateException: The new owner clevercloud-pulsar-broker-c3-n1:8443 is inactive. Caused by: java.lang.IllegalStateException: The new owner clevercloud-pulsar-broker-c3-n1:8443 is inactive.
```

Since the monitoring didn’t catch it, the broker was up so the rolling process continued. The more it started, the more they tried to reach the topic metadata, fulfilling the *Lookup requests queue* :

```shell{linenos=table,linenostart=1}
```shell
2024-08-02T08:49:12,448+0000 [pulsar-io-3-5] WARN org.apache.pulsar.client.impl.MultiTopicsConsumerImpl - Failed to get partitions for topic to determine if new partitions are added
java.util.concurrent.CompletionException: org.apache.pulsar.client.api.PulsarClientException$TooManyRequestsException: {"errorMsg":"Failed due to too many pending lookup requests","reqId":1946266099531305237, "remote":"clevercloud-pulsar-broker-c3-n9/192.168.2.9:6650", "local":"/192.168.2.9:58654"} Caused by: org.apache.pulsar.client.api.PulsarClientException$TooManyRequestsException: {"errorMsg":"Failed due to too many pending lookup requests","reqId":1946266099531305237, "remote":"clevercloud-pulsar-broker-c3-n9/192.168.2.9:6650", "local":"/192.168.2.9:58654"}
```
Ultimately, the topic has never been properly created and brokers ended up in a state where no topics were loaded :
```shell{linenos=table,linenostart=1}
```shell
2024-08-02T09:16:23,828+0000 [pulsar-io-3-7] WARN org.apache.pulsar.client.impl.BinaryProtoLookupService - [non-persistent://pulsar/system/loadbalancer-broker-load-data] failed to get Partitioned metadata : {"errorMsg":"Failed to get partition metadata","reqId":2658063197115238188, "remote":"clevercloud-pulsar-broker-c3-n1/192.168.2.1:6650", "local":"/192.168.2.1:34522"} org.apache.pulsar.client.api.PulsarClientException$BrokerMetadataException: {"errorMsg":"Failed to get partition metadata","reqId":2658063197115238188, "remote":"clevercloud-pulsar-broker-c3-n1/192.168.2.1:6650", "local":"/192.168.2.1:34522"}
2024-08-02T09:16:24,346+0000 [broker-client-shared-scheduled-executor-7-1] WARN org.apache.pulsar.client.impl.PulsarClientImpl - [topic: non-persistent://pulsar/system/loadbalancer-broker-load-data] Could not get connection while getPartitionedTopicMetadata -- Will try again in 795 ms
2024-08-02T09:16:24,348+0000 [pulsar-io-3-15] ERROR org.apache.pulsar.broker.service.ServerCnx - non-persistent://pulsar/system/loadbalancer-broker-load-data [id: 0xa5782645, L:/192.168.2.1:6650 - R:/192.168.2.1:34522] [SR:192.168.2.1, state:Connected] Failed to get partition metadata java.util.concurrent.CompletionException: java.lang.IllegalStateException: The registry already closed.Caused by: java.lang.IllegalStateException: The registry already closed.
Expand Down

0 comments on commit 9041093

Please sign in to comment.