Extremely slow Nominatim install with Docker

Question

I am trying to install Nominatim based off of a dockerized version of the tool found here: https://github.com/mediagis/nominatim-docker. I was successful with installing dockerised Nominatim with a smaller country locally. And the process ran pretty fast.

Then, I got an EC2 instance with, 64GB memory, 1TB SSD, and 6 core processors to do an install with the entire planet (48GB). The most of the planet file got processed fast, in about 6hours and when processing the relations gave an error and failed. When I ran the planet file based on the same docker image, the object parsing (specially ways) were extremely slow. About 100 times slower than the initial run. This is the error I got when the planet ran faster:

Processing: Node(5640116k 412.0k/s) Way(625947k 118.89k/s) Relation(628790 712.11/s)2019-12-18 22:48:23.270 UTC [210] LOG:  incomplete message from client
2019-12-18 22:48:23.270 UTC [210] CONTEXT:  COPY place, line 233940695
2019-12-18 22:48:23.270 UTC [210] STATEMENT:  COPY place FROM STDIN
2019-12-18 22:48:23.270 UTC [207] LOG:  incomplete message from client
2019-12-18 22:48:23.270 UTC [207] CONTEXT:  COPY planet_osm_rels, line 598784
2019-12-18 22:48:23.270 UTC [207] STATEMENT:  COPY planet_osm_rels FROM STDIN
Killed
2019-12-18 22:48:23.275 UTC [210] ERROR:  unexpected EOF on client connection with an open transaction
2019-12-18 22:48:23.275 UTC [210] CONTEXT:  COPY place, line 233940695
2019-12-18 22:48:23.275 UTC [210] STATEMENT:  COPY place FROM STDIN
2019-12-18 22:48:23.275 UTC [207] ERROR:  unexpected EOF on client connection with an open transaction
2019-12-18 22:48:23.275 UTC [207] CONTEXT:  COPY planet_osm_rels, line 598784
2019-12-18 22:48:23.275 UTC [207] STATEMENT:  COPY planet_osm_rels FROM STDIN
2019-12-18 22:48:23.275 UTC [207] LOG:  could not send data to client: Broken pipe
2019-12-18 22:48:23.275 UTC [207] STATEMENT:  COPY planet_osm_rels FROM STDIN
2019-12-18 22:48:23.275 UTC [210] LOG:  could not send data to client: Broken pipe
2019-12-18 22:48:23.275 UTC [210] STATEMENT:  COPY place FROM STDIN
2019-12-18 22:48:23.275 UTC [207] FATAL:  terminating connection because protocol synchronization was lost
2019-12-18 22:48:23.325 UTC [210] FATAL:  terminating connection because protocol synchronization was lost
ERROR: No Data
string(7) "No Data"

Finally, I just took a pbf of North America (8GB) and tried the install. Object parsing related to North America went relatively fast, but it is still working on rankings and indexes. The install is going on for over 16 hours.

I didn't do any PostgreSQL optimizations before running the install. When the object parsing is slow, I see the container is only using about 1% of the available memory.

In Nominatim install instructions (here: https://nominatim.org/release-docs/latest/admin/Installation/) I see that with less config (32 GB RAM) a planet install is taking only 2 days. What am I doing wrong and how can I optimize the process?

Accepted Answer

The mediagis docker image is not particularly well suited for large size imports like North America or the planet. Read the original Nominatim installation instructions to get an idea about tuning and here in particular the section on PostgreSQL tuning, using flatnode files and notes on time and memory use during import.

A machine of your size should manage a planet import in 2-3 days but only if you use flatnode files and a tuned postgres configuration.

Answer 2

In the Docker file I don't see "--osm2pgsql-cache" set for the setup.php (https://nominatim.org/release-docs/latest/admin/Import-and-Update/#initial-import-of-the-data) That will give osm2pgsql more RAM to work with.

Extremely slow Nominatim install with Docker

Related questions