-
-
Notifications
You must be signed in to change notification settings - Fork 198
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
WATCH_MODE info #288
Comments
If this is on road map,can i try this? |
Hi @sharkAndshark, Sure! |
Based on the discusion in #349 ,there will be no watch mode? @stepankuzmin |
@sharkAndshark I think we may need to rethink the watch mode, and possibly add it back before the next release.
That said, I really feel like the more substantial changes should be done with kubernetes and helm and deployments, and the
|
@nyurik I will give it a a shot and then more discussion |
Sounds good! @stepankuzmin any thoughts? |
@sharkAndshark sure, that would be great 👍 |
@nyurik We are currently using the latest version of martin, and there are some real benefits in using this version. |
@pascal-codetaal I'm beginning to think we really do need to reimplement it, and some help would be great. So lets have a dedicated My only concern is that we would want to avoid a global lock on each request. I'm not yet certain how to do that best. |
@pascal-codetaal
there is no downtime. kube will ensure the old pod(s) is/are working while the new one gets deployed. Just make sure you have enough capacity on the node to spin a new deployment. Also, you should ideally have more(>1) replicas. I have the same issues and being under time pressure I resorted to something else An env vars specify how often the url/config is pulled and where the URL is pulled from. This might not be very elegant but it works like a charm specially because i use a tool to create (optionally upload to azure) the config files. We actually update the layers on a daily, sometimes hourly basis. In future i will pool the remote config once a day. The tool has a quirk that it creates configs only for tables that have the "publish=True" in the table description. The same is valid for columns, if you mark a column with publish=False the toll will skip it. |
We don't have a downtime, it just can take up to 50 seconds before the new pod is up and running... |
I would love to be able to reload the schema config without a full restart. The PostgREST project has two cool ways to do this: https://postgrest.org/en/stable/schema_cache.html#schema-cache-reloading
The second method in particular is interesting because it allows the DB admin to set up custom triggers that would send notifications to the custom channel any time a geospatial table changes. |
There are currently two issues I see (i'm sure there are more):
|
We just ran into this issue after switching from pg_tileserv to Martin. With pg_tileserv all new tables where discovered right away. Now, we have to restart our Martin Docker container to get an updated table list. Our docker setup https://github.com/FixMyBerlin/atlas-geo/blob/main/docker-compose.yml starts our backend server with db, processing and tiles. It looks like we don't have a good way to restart the tile docker after the app image closed and finished processing our data… Is there some API like |
@tordans we all want that api to happen - I plan to work on it in the future, but no certainty on the timeframe -- it is mostly a personal project, so if anyone has any time to work on it, I will be happy to guide and assist that effort. |
Hello @nyurik and the Martin contributors, We've been closely following the discussion about the need for a watch mode or similar feature to enable dynamic config reloading in Martin. As we understand the significance of this feature for users, and us, who need to see updated data without substantial downtime, we're keen on contributing to its development. Though we're not currently well-versed in Rust, we're ready to put in the necessary time to become proficient and contribute meaningfully to this project. We're particularly interested in understanding the preferred approach. Could you provide us with some guidance on where you believe we should start, or if there are any particular areas of the codebase or documentation that we should familiarize ourselves with first? We're looking forward to collaborating with the community on this feature. Thank you for considering our offer to help. |
@pascal-codetaal I will be happy to help you. The very first thing I think we should evaluate is the performance aspect outlined in #288 (comment) -- create a tiny Actix app using one of the examples they have as a starting point. Inside that app, create two endpoints, e.g. This should get you familiarized with the Rust ecosystem, making it much easier to make changes to a bigger project. #[derive(Clone)]
struct State {
pub values: HashMap<String, String>,
pub values_mutex: Arc<Mutex<HashMap<String, String>>>,
}
impl State {
pub fn new() -> Self {
let values = vec![(String::from("key"), String::from("value"))]
.into_iter()
.collect::<HashMap<String, String>>();
Self {
values_mutex: Arc::new(Mutex::new(values.clone())),
values,
}
}
}
#[get("/lock")]
async fn get_lock(state: Data<State>) -> String {
let vals = state.values_mutex.lock().unwrap();
vals.get("key").unwrap().clone()
}
#[get("/nolock")]
async fn get_nolock(state: Data<State>) -> String {
state.values.get("key").unwrap().clone()
} |
@nyurik the development will be done by @mesudBe-Orbit |
@nyurik @pascal-codetaal the development is complete, and you can access the tiny application and view the results by following this link: https://github.com/mesudBe-Orbit/actix-lock-stress-tester?tab=readme-ov-file#test-results |
awesome news, thanks! So this should make it easier. The current config system process:
The Line 115 in a864b99
PostgreSQL resolution is much more involved than pmtiles/mbtiles. It runs these steps in parallel for all configured postgres connections:
martin/martin/src/pg/config.rs Line 116 in a864b99
So one path to implement watch mode:
|
Please run your code with |
@nyurik the 5 minute run had the same result for both endpoints.
|
@pascal-codetaal sounds good. One other thing - I would recommend creating a Martin fork, create a branch there, and create a |
@nyurik we have created a draft PR into martin. you can find it here And commit and push our latest code (the latest commit was about mistakenly added repetition). |
@mesudBe-Orbit yes, thanks, i saw it and already commented on that PR. Keep adding things to it (by pushing to the same branch), and once it is ready to be merged, click the |
pmtiles support for this would be good as well. I outlined how it can be done in stadiamaps/pmtiles-rs#40 - and once done, we can use it in Martin too. |
Hello there
I'd like to know if setting WATCH_MODE=true makes martin reload the config file.
I am publishing many postgis layer with even more properties/columns and in order to limit these i need to use a configuration file.
I am thinking to create a python script that will generate the config yaml and it would be really nice if martin could re-read this in WATCH_MODE
thank you for this great software box
The text was updated successfully, but these errors were encountered: