Skip to content
GitLab
Open
Issue created by Administrator@rootContributor4 of 4 checklist items completed4/4 checklist items

[BUG] Slow deserialisation into objects in generated python sdk

Created by: rizwansaeed

Bug Report Checklist

  • Have you provided a full/minimal spec to reproduce the issue?
  • Have you validated the input using an OpenAPI validator (example)?
  • Have you tested with the latest master to confirm the issue still exists?
  • Have you searched for related issues/PRs?

Related Issues: https://github.com/OpenAPITools/openapi-generator/issues/4181

Description

We are seeing slow performance when deserialising responses from API calls into model objects. The actual deserialisation from JSON to dict looks ok, but mapping the dict to model objects is slow

openapi-generator version

5.1.0

OpenAPI declaration file content or url

https://www.lusid.com/api/swagger/v0/swagger.json

Generation Details
java -jar openapi-generator-cli.jar generate \
    -i $swagger_file \
    -g python-legacy
Steps to reproduce

A sample repo to reproduce can be found at https://github.com/rizwansaeed/OpenAPIPerf, this uses the generated sdk which also be found at https://github.com/finbourne/lusid-sdk-python-preview

/data/lusid.json - OpenAPI spec /data/response.json - sample response /python/main.py - test to show the deserialisation performance

With this we see it takes ~ 24s to deserialise the response into model objects

         47218117 function calls (45073006 primitive calls) in 24.371 seconds

   Ordered by: cumulative time

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
        1    0.000    0.000   24.371   24.371 /OpenAPIPerf/python/venv/lib/python3.9/site-packages/lusid/api/transaction_portfolios_api.py:3257(get_transactions)
        1    0.000    0.000   24.371   24.371 /OpenAPIPerf/python/venv/lib/python3.9/site-packages/lusid/api/transaction_portfolios_api.py:3303(get_transactions_with_http_info)
        1    0.000    0.000   24.371   24.371 /OpenAPIPerf/python/venv/lib/python3.9/site-packages/lusid/api_client.py:327(call_api)
        1    0.013    0.013   24.371   24.371 /OpenAPIPerf/python/venv/lib/python3.9/site-packages/lusid/api_client.py:121(__call_api)
        1    0.000    0.000   24.342   24.342 /OpenAPIPerf/python/venv/lib/python3.9/site-packages/lusid/api_client.py:266(deserialize)
 585025/1    0.948    0.000   24.061   24.061 /OpenAPIPerf/python/venv/lib/python3.9/site-packages/lusid/api_client.py:288(__deserialize)
 260006/1    1.968    0.000   24.061   24.061 /OpenAPIPerf/python/venv/lib/python3.9/site-packages/lusid/api_client.py:664(__deserialize_model)
        2    0.007    0.003   24.054   12.027 /OpenAPIPerf/python/venv/lib/python3.9/site-packages/lusid/api_client.py:302(<listcomp>)
    10000    0.105    0.000   18.360    0.002 /OpenAPIPerf/python/venv/lib/python3.9/site-packages/lusid/api_client.py:307(<dictcomp>)
   260006    0.160    0.000   15.923    0.000 /OpenAPIPerf/python/venv/lib/python3.9/site-packages/lusid/configuration.py:245(get_default_copy)
   260006    1.950    0.000   15.763    0.000 /OpenAPIPerf/python/venv/lib/python3.9/site-packages/lusid/configuration.py:90(__init__)
9880228/8580198    3.630    0.000   11.581    0.000 /OpenAPIPerf/python/venv/lib/python3.9/site-packages/lusid/configuration.py:224(__setattr__)
   120000    0.211    0.000    7.601    0.000 /OpenAPIPerf/python/venv/lib/python3.9/site-packages/lusid/models/perpetual_property.py:56(__init__)
   120000    0.169    0.000    7.347    0.000 /OpenAPIPerf/python/venv/lib/python3.9/site-packages/lusid/models/property_value.py:59(__init__)
   260006    0.525    0.000    6.506    0.000 /OpenAPIPerf/python/venv/lib/python3.9/site-packages/lusid/configuration.py:299(debug)
   520012    0.347    0.000    4.970    0.000 /usr/local/Cellar/python@3.9/3.9.8/Frameworks/Python.framework/Versions/3.9/lib/python3.9/logging/__init__.py:1417(setLevel)
   520012    2.748    0.000    4.440    0.000 /usr/local/Cellar/python@3.9/3.9.8/Frameworks/Python.framework/Versions/3.9/lib/python3.9/logging/__init__.py:1372(_clear_cache)
    15002    0.012    0.000    3.460    0.000 /OpenAPIPerf/python/venv/lib/python3.9/site-packages/lusid/api_client.py:643(__deserialize_datetime)

I have tried this with the latest python (non-legacy) version and observe similar performance characteristics

Process finished with exit code 0
         52273648 function calls (46098366 primitive calls) in 25.016 seconds

   Ordered by: cumulative time

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
        1    0.000    0.000   25.016   25.016 /lusid-sdk-python-preview/sdk/lusid/api_client.py:761(__call__)
        1    0.000    0.000   25.016   25.016 /lusid-sdk-python-preview/sdk/lusid/api/transaction_portfolios_api.py:3039(__get_transactions)
        1    0.000    0.000   25.016   25.016 /lusid-sdk-python-preview/sdk/lusid/api_client.py:774(call_with_http_info)
        1    0.000    0.000   25.015   25.015 /lusid-sdk-python-preview/sdk/lusid/api_client.py:335(call_api)
        1    0.000    0.000   25.015   25.015 /lusid-sdk-python-preview/sdk/lusid/api_client.py:118(__call_api)
        1    0.000    0.000   25.008   25.008 /lusid-sdk-python-preview/sdk/lusid/api_client.py:290(deserialize)
585034/10    1.214    0.000   24.710    2.471 /lusid-sdk-python-preview/sdk/lusid/model_utils.py:1357(validate_and_convert_types)
 275008/1    0.476    0.000   24.710   24.710 /lusid-sdk-python-preview/sdk/lusid/model_utils.py:1244(attempt_convert_item)
 260006/1    0.926    0.000   24.710   24.710 /lusid-sdk-python-preview/sdk/lusid/model_utils.py:1166(deserialize_model)
 260006/1    0.634    0.000   24.710   24.710 /lusid-sdk-python-preview/sdk/lusid/model_utils.py:1562(wrapped_init)
        1    0.000    0.000   24.710   24.710 /lusid-sdk-python-preview/sdk/lusid/model/versioned_resource_list_of_transaction.py:118(__init__)
2015056/11    0.658    0.000   24.710    2.246 /lusid-sdk-python-preview/sdk/lusid/model_utils.py:166(__setattr__)
2015056/11    0.935    0.000   24.710    2.246 /lusid-sdk-python-preview/sdk/lusid/model_utils.py:347(__setitem__)
 455020/5    3.066    0.000   24.710    4.942 /lusid-sdk-python-preview/sdk/lusid/model_utils.py:106(set_attribute)
     5000    0.061    0.000   24.422    0.005 /lusid-sdk-python-preview/sdk/lusid/model/transaction.py:142(__init__)
480045/170034    0.192    0.000   21.073    0.000 {built-in method builtins.setattr}
   120000    0.474    0.000   13.597    0.000 /lusid-sdk-python-preview/sdk/lusid/model/perpetual_property.py:106(__init__)
   120000    0.447    0.000    5.855    0.000 /lusid-sdk-python-preview/sdk/lusid/model/property_value.py:110(__init__)
  1590070    1.895    0.000    4.245    0.000 /lusid-sdk-python-preview/sdk/lusid/model_utils.py:621(get_simple_class)
    15002    0.046    0.000    3.582    0.000 /lusid-sdk-python-preview/sdk/lusid/model_utils.py:1062(deserialize_primitive)

For reference, and whilst it is not an exact like for like comparison, the test in csharp/OpenAPIPerf/UnitTest1.cs deserialising the model objects in less than 1 second.

Suggest a fix

It looks like when the __deserialise function comes across a list or dictionary it sequentially iterates over each item, and when there are nested structured objects this exacerbates the issue.

Are there any quick wins/settings we can use to improve the performance?