Nominatim Setup with latest Planet PBF always fails

Hi, in the last few days I have made several attempts to set up Nominatim with the latest planet pbf but the import always failed after it already ran for quite some time. Attempts were made with AWS Ubuntu 18.04 AMI, 1TB of Storage and both a m5.2xlarge (32GB memory + 32GB swapfile) as well as a m5.4xlarge (64GB memory, no swapfile). Using smaller pbf's like germany works fine.

I am always following the installation guide on nominatim.org with the exception that I am using a different user and directory: export USERNAME=ubuntu export USERHOME=/home/ubuntu/nominatim

For my latest try (32GB memory + 32GB swapfile) I used this command: ./utils/setup.php --osm-file planet-latest.osm.pbf --all --osm2pgsql-cache 24000 2>&1 | tee setup.log

and it failed with this message: Processing: Node(5054999k 477.8k/s) Way(564535k 25.40k/s) Relation(3376050 729.64/s) ERROR: 1 select * from place limit 1 [nativecode=ERROR: relation "place" does not exist LINE 1: select * from place limit 1 ^]

There are no errors in /var/log/syslog.

PostgreSQL config: shared_buffers = 2GB maintenance_work_mem = 10GB work_mem = 50MB effective_cache_size = 24GB synchronous_commit = off checkpoint_timeout = 10min checkpoint_completion_target = 0.9

Excerpt from the postgresql log: 2019-03-02 21:07:12.612 UTC [20827] LOG: checkpoints are occurring too frequently (23 seconds apart) 2019-03-02 21:07:12.612 UTC [20827] HINT: Consider increasing the configuration parameter "max_wal_size". 2019-03-02 21:07:41.033 UTC [20827] LOG: checkpoints are occurring too frequently (29 seconds apart) 2019-03-02 21:07:41.033 UTC [20827] HINT: Consider increasing the configuration parameter "max_wal_size". 2019-03-02 21:09:53.090 UTC [20827] LOG: checkpoints are occurring too frequently (29 seconds apart) 2019-03-02 21:09:53.090 UTC [20827] HINT: Consider increasing the configuration parameter "max_wal_size". 2019-03-02 22:32:11.844 UTC [21697] ubuntu@nominatim ERROR: relation "place" does not exist at character 15 2019-03-02 22:32:11.844 UTC [21697] ubuntu@nominatim STATEMENT: select * from place limit 1 2019-03-02 22:32:14.486 UTC [21805] ubuntu@nominatim LOG: incomplete message from client 2019-03-02 22:32:14.486 UTC [21805] ubuntu@nominatim CONTEXT: COPY place, line 212096963 2019-03-02 22:32:14.486 UTC [21805] ubuntu@nominatim STATEMENT: COPY place (osm_type, osm_id, class, type, name, admin_level, address, extratags, geometry) FROM STDIN 2019-03-02 22:32:14.486 UTC [21805] ubuntu@nominatim ERROR: unexpected EOF on client connection with an open transaction 2019-03-02 22:32:14.486 UTC [21805] ubuntu@nominatim CONTEXT: COPY place, line 212096963 2019-03-02 22:32:14.486 UTC [21805] ubuntu@nominatim STATEMENT: COPY place (osm_type, osm_id, class, type, name, admin_level, address, extratags, geometry) FROM STDIN 2019-03-02 22:32:14.486 UTC [21805] ubuntu@nominatim LOG: could not send data to client: Broken pipe 2019-03-02 22:32:14.486 UTC [21805] ubuntu@nominatim STATEMENT: COPY place (osm_type, osm_id, class, type, name, admin_level, address, extratags, geometry) FROM STDIN 2019-03-02 22:32:14.487 UTC [21808] ubuntu@nominatim LOG: incomplete message from client 2019-03-02 22:32:14.487 UTC [21808] ubuntu@nominatim CONTEXT: COPY planet_osm_rels, line 3376543 2019-03-02 22:32:14.487 UTC [21808] ubuntu@nominatim STATEMENT: COPY planet_osm_rels FROM STDIN;


2019-03-02 22:32:14.487 UTC [21808] ubuntu@nominatim ERROR:  unexpected EOF on client connection with an open transaction
2019-03-02 22:32:14.487 UTC [21808] ubuntu@nominatim CONTEXT:  COPY planet_osm_rels, line 3376543
2019-03-02 22:32:14.487 UTC [21808] ubuntu@nominatim STATEMENT:  COPY planet_osm_rels FROM STDIN;
2019-03-02 22:32:14.487 UTC [21808] ubuntu@nominatim LOG:  could not send data to client: Broken pipe
2019-03-02 22:32:14.487 UTC [21808] ubuntu@nominatim STATEMENT:  COPY planet_osm_rels FROM STDIN;

2019-03-02 22:32:14.487 UTC [21806] ubuntu@nominatim LOG: unexpected EOF on client connection with an open transaction 2019-03-02 22:32:14.487 UTC [21807] ubuntu@nominatim LOG: unexpected EOF on client connection with an open transaction 2019-03-02 22:32:14.520 UTC [21808] ubuntu@nominatim FATAL: terminating connection because protocol synchronization was lost 2019-03-02 22:32:14.607 UTC [21805] ubuntu@nominatim FATAL: terminating connection because protocol synchronization was lost

Is there any obvious error I am making?

Edit: I tried using a flatnode file a suggested by mtmail, but got the same result as before. The last two runs exited with the same error but already failed while still processing the nodes.

Here is the full setup script I am using: setup.sh

asked 03 Mar '19, 00:04

Greenish Purple
11●1●1●4
accept rate: 0%

edited 03 Mar '19, 08:47

One Answer:

The processing line for whole planet would go to 6, possible 7 million relations (OpenStreetMap data keeps growing every month).

The setup steps and setting you posted look alright. 32GB should work, or rather the failure would look differently if you ran out of memory in that step of the setup. My last import used a cache size of 28000.

To me it all points to the client (osm2pgsql)-server connection crashing hard. Those machines have network-mounted drives. Which https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/EBSVolumeTypes.html do you use? Is it possible you hit a IOPs burst limit? Any AWS Cloudwatch metrics that would show drive throughput or IOPs used? I've seen the postgresql server die and missing network connection (if installed on a separate server) but those have different log lines in the postgresql log.

Apart from trying again, or again with higher spec drives, I suggest using the flatnode file. Maybe you already do, the output posted here is to short to tell. It's explained on https://nominatim.org/release-docs/latest/admin/Import-and-Update/

permanent link

answered 03 Mar '19, 01:10

mtmail
3.8k●10●59
accept rate: 30%

Thanks for taking the time to help me out. I am using a general purpose SSD that has a performance of 3000 IOPS and about all of it is used during the db import.

Does using the flatnode file mean a hit to performance? I am mostly interested in the reverse geocode lookup.

Edit: I'm currently trying both just running it again as well as using flatnode and will look at how both servers fared tomorrow.

(03 Mar '19, 01:34) Greenish Purple

Both setups failed quite early this time, using a flatnodes file made no difference. I added my full installation script to the question, maybe there is an error in there I don't see.

(03 Mar '19, 08:51) Greenish Purple

I found something in the osm2pgsql issue list https://github.com/openstreetmap/osm2pgsql/issues/159 which points to database looses connection and a user reports more RAM helped. Here https://github.com/openstreetmap/osm2pgsql/issues/855 connecting to the database via unix socket instead of port was recommended. Lastly just today another user reported the same issue just hours ago https://github.com/openstreetmap/Nominatim/issues/1320

(03 Mar '19, 12:37) mtmail

Okay, I am going to modify the setup script to omit the port, which should mean that the connection is established via socket instead. I will also spin up an even bigger server with 192GB memory and try that with the unmodified setup script.

Thank you for spending the time to dig out possible solutions :)

(03 Mar '19, 14:37) Greenish Purple

Result: both 32GB memory + 32GB swapfile with socket connection to postgres and the 192GB memory server failed with the same error message as above, logfiles also look the same :(

Are there any logs I can dig through to maybe get a better error message besides the postgres log?

(03 Mar '19, 16:59) Greenish Purple

Markdown Basics

*italic* or _italic_
**bold** or __bold__
link:[text](http://url.com/ "title")
image?![alt text](/path/img.jpg "title")
numbered list: 1. Foo 2. Bar
to add a line break simply add two spaces to where you would like the new line to be.
basic HTML tags are also supported

learn more about Markdown

Question tags:

nominatim ×591
installation ×55
planet ×53
pbf ×51

question asked: 03 Mar '19, 00:04

question was seen: 1,045 times

last updated: 03 Mar '19, 16:59

Nominatim Setup with latest Planet PBF always fails

Related questions