Jürgen Baier wrote:

Hi,

I'm using LMDB for mapping MD5 hash codes to some data. I noticed that a virtualized environment (Xen/Windows on our own servers and AWS/Windows) slows down LMDB significantly (e.g. a certain workload is executed in 30 minutes on a local machine vs. 15 hours on AWS).

Sounds like whatever filesystem is running on AWS has a really poor implementation of mmap.

My application also has an implementation in SQLite which is slower than LMDB on a local machine. But its performance is similar on AWS. The rough numbers are something like

Local:

• LMDB: 30 minutes
• SQLite: 45 minutes

AWS:

• LMDB: 15 hours
• SQLite: 50 minutes

I wanted to ask if there is something I can do to speed up LMDB on AWS? Maybe I'm missing something obvious.

Are reads slow, or writes? Or both? If writes are slow, try setting MDB_NOSYNC. If reads are slow,\ use a different filesystem.

Thanks,

Jürgen

...

...

Jürgen Baier baier@semedy.com schrieb am 07.10.2021 um 08:07 in

Nachricht a83eb812-351f-b332-45d0-3ef74b9c5c30@semedy.com:

Hi,

I'm using LMDB for mapping MD5 hash codes to some data. I noticed that a virtualized environment (Xen/Windows on our own servers and AWS/Windows) slows down LMDB significantly (e.g. a certain workload is executed in 30 minutes on a local machine vs. 15 hours on AWS).

I don't know whether Winows uses huge pages at all, but AFAIK huge pages are unavaiable under Xen. Maybe that makes a difference for large memory configurations (in addition to the other things mentioned). The other thing is whether your VM has a dedicated CPU or an "overcommitted" shared CPU.

However 15 hours vs. 30 minutes sounds drastic.

My application also has an implementation in SQLite which is slower than LMDB on a local machine. But its performance is similar on AWS. The rough numbers are something like

Local:

• LMDB: 30 minutes
• SQLite: 45 minutes

AWS:

• LMDB: 15 hours
• SQLite: 50 minutes

I wanted to ask if there is something I can do to speed up LMDB on AWS? Maybe I'm missing something obvious.

Thanks,

Jürgen

-- Juergen Baier