40
19

When looking at the main map on OpenStreetMap.org, some of the data that was added only minutes ago already shows up, whereas other data that was added weeks or months ago isn't visible yet. How does the update process of the main mapnik map work?

asked 14 Jul '10, 14:00

apmon's gravatar image

apmon
6.3k184456
accept rate: 20%

edited 18 Oct '13, 03:14

aseerel4c26's gravatar image

aseerel4c26 ♦
32.2k16239552


One Answer:
88

The short answer to this question is: The rendering of the main (mapnik) map should typically update within a few (tens of) minutes, however occasionally it can take longer.

The longer answer is a little more complicated, as there are many components that interact in subtle ways to produce the updated results. So I'll try and explain the process, which may be a little technical at points:

First of all, if you edit OpenStreetMap data, your changes get applied to the main database and are immediately visible from the API. The rendering of the map however uses a secondary database that needs to be kept updated from the main database. For this purpose, the main database publishes minutely diffs that are then applied to the rendering database on OpenStreetMap.org and other rendering servers. As the name suggests, these diffs typically are produced once a minute and so the rendering database is typically 1 - 2 minutes behind the main database. Occasionally (although this shouldn't happen) the diffs get stuck though, and this delay may increase a bit. You can see graphs the current delay of imports for the two rendering servers, yevaud and orm.

Once the new data is in the rendering database, it then needs to actually get rendered. However, the server does not have the capacity and resources to rerender all changed tiles immediately, so a clever algorithm tries to only render those tiles that are necessary in an order to try and ensure that the map appears to be as up to date as possible:

When your browser requests a tile from the server, the server checks if it has already rendered an up to date copy of this tile. If it has, then it simply sends back that copy to the client. If the copy the server has on the other hand is found to be out of date, it checks to see if it (the server) is not overloaded (e.g. in peak time) then tries to rerender the tile from the current rendering database and send this back to the client. If however it is overloaded, it returns the out of date copy back to the client. Similarly, if the rendering process takes more than 3 seconds, again the old tile gets returned to the client to not make the browser wait for ages to get the map. In both cases though, the tile gets added to the rendering queue and will typically get rendered in the background in the next hour or so. Therefore if you visit the same area e.g. an hour later, it should then have finished the rendering and you will see the up to date data.

In some cases when the server is severely overloaded, the rendering queue might fill up and then the rendering request may get dropped, which means the area won't get rerendered and updated until someone else goes and views the area when the queues are no longer full, in which case the procedure above applies again.

You can see if the rendering queue is full and thus it is likely that the map currently won't appear to update on these graphs.

Now this leaves the question of how and when does the server decide if a tile is out of date or not: For this purpose, during the import of the diffs into the rendering db, an extra program calculates which tiles need rerendering based on those diffs. As deciding which tiles are effected, is a hard problem without actually rendering everything, this is only based on heuristics. I.e. it is based upon if there are nodes in the tile, or ways that are in the diff have nodes in those tiles. Relations currently get ignored completely in this process. Similarly, for areas, only the perimeter of the area gets expired.

If you suspect that the tile was not correctly expired, there is a way to manually overwrite this expiry mechanism by appending a /dirty to the url which will append the tile to the back of the rendering queue. However, in nearly all cases this should not be necessary.

As tiles only get rerendered once they are viewed, some of the higher zoom levels that are seldomly viewed might therefore appear out of date by quite a bit. It also means that neighbouring tiles that might have been viewed recently are more up to date, and thus show some strange artefacts of different rendering times.

In most cases hitting the refresh button a couple of times a few minutes apart will fix these issues.

On top of all this, is the additional issue of (proxy) caching, both in your browser and server side to reduce load. This may introduce an additional delay of a few hours and may cause the tile not to get rerenderd as the request might never reaches the server to trigger the rerender.

As you see, this whole things is complicated, but in most cases the short answer is sufficient. i.e. it normally updates within a few minutes.

permanent link

answered 14 Jul '10, 14:41

apmon's gravatar image

apmon
6.3k184456
accept rate: 20%

edited 20 Oct '13, 07:07

pnorman's gravatar image

pnorman
2.4k52140

7

apmon excellent explanation, but it would benefit from a small additional mention that coastlines are an exception. Coastlines being updated every so often. (typically twice a month.)

(20 Aug '10, 12:30) Firefishy
2

more information about the coastline process in the wiki: http://wiki.openstreetmap.org/wiki/Coastline#Main_Mapnik_layer

(18 Sep '11, 16:40) dieterdreist
1

Seems like there is heavy daytime load dependency to be seen in the graphs. What about not dropping any tiles from rendering but instead moving them to rendering during night off-peak hours? Finally, higher server load is not a big financial issues, right? I mean in terms of higher electricity comsumption for instance...

(26 Feb '12, 10:14) Kozuch

I can see my changes are being applied to the standard map, but the open cyclemap layer remains unchanged. How often does the open cyclemap get updated? I have been making corrections to some cycle paths that are not connected correctly thus causing routing errors.

(02 May '12, 11:24) paulcooke
1

The cycle map layer uses Andy Allen's opencyclemap tiles. He says that "The map is updated every week (on a Thursday or Friday, normally)".

(02 May '12, 12:23) SomeoneElse ♦
2

Does the above described algorithm only apply for zoomlevels >=13 ? I usually always have to manually /dirty tiles with zoomlevel 12, 11 and 10 to have tertiary roads show up on those tiles...

(23 Jan '14, 12:58) katpatuka

What I read above seems quite clear but, what happens to me is a little more strange: I deleted an old obsolete track segment and I drew a new track more actual replacing the old one lightly displaced with respect the old one. I did all the modifications about 24 hours ago. Now I can see the updated situation if I set the zoom to have the map scale on 300m or 100m. If the map scale is on 50m the modification is only partial If the map scale is 30m no modification is shown and I see the old situation If the map scale is 20m the modifications are completely shown All these things happen using the Standard Map; if I set the Cycling Map no modification appears on every zoom level.

(07 Jan '17, 17:25) gbal45
1

@gbal45 pls do not add further questions to existing question and pls don't add them as answers. In any case you are simply seeing in browser caching effects. Refreshing and/or deleting the cache should help except if there is an overload situation on the tile servers.

(07 Jan '17, 20:07) SimonPoole ♦

Thanks for the great answer!

Just a heads up, the http://munin.openstreetmap.org/openstreetmap/yevaud.openstreetmap/#renderd link is broken, and returns a 404 error.

(11 Jan '17, 02:21) keithonearth
1

Additional info: zoom levels 12 and below do not get marked dirty by map changes. Apparently they only re-render manually or when render styles change.

(02 Apr '17, 15:20) MrPete
2

Some additional details:

(18 Dec '17, 03:59) kocio
showing 5 of 11 show 6 more comments

Markdown Basics

  • *italic* or _italic_
  • **bold** or __bold__
  • link:[text](http://url.com/ "title")
  • image?![alt text](/path/img.jpg "title")
  • numbered list: 1. Foo 2. Bar
  • to add a line break simply add two spaces to where you would like the new line to be.
  • basic HTML tags are also supported

Question tags:

×443
×396
×173
×7

question asked: 14 Jul '10, 14:00

question was seen: 39,464 times

last updated: 30 Oct '18, 12:26

powered by OSQA