Skip to content
GitLab
Projects Groups Snippets
  • /
  • Help
    • Help
    • Support
    • Community forum
    • Submit feedback
    • Contribute to GitLab
  • Sign in / Register
  • B buck
  • Project information
    • Project information
    • Activity
    • Labels
    • Members
  • Repository
    • Repository
    • Files
    • Commits
    • Branches
    • Tags
    • Contributors
    • Graph
    • Compare
  • Issues 201
    • Issues 201
    • List
    • Boards
    • Service Desk
    • Milestones
  • Merge requests 22
    • Merge requests 22
  • CI/CD
    • CI/CD
    • Pipelines
    • Jobs
    • Schedules
  • Deployments
    • Deployments
    • Environments
    • Releases
  • Packages and registries
    • Packages and registries
    • Package Registry
    • Infrastructure Registry
  • Monitor
    • Monitor
    • Incidents
  • Analytics
    • Analytics
    • Value stream
    • CI/CD
    • Repository
  • Wiki
    • Wiki
  • Snippets
    • Snippets
  • Activity
  • Graph
  • Create a new issue
  • Jobs
  • Commits
  • Issue Boards
Collapse sidebar
  • Meta
  • buck
  • Merge requests
  • !2547

Add support for async worker processes

  • Review changes

  • Download
  • Email patches
  • Plain diff
Merged Mike Kaplinskiy requested to merge github/fork/mikekap/async-worker-process into master Sep 25, 2020
  • Overview 21
  • Commits 7
  • Pipelines 0
  • Changes 26

The idea is that you might be able to parallelize internally in the worker process, with the benefit of shared memory to cache details about dependencies. Async worker processes can receive & run multiple commands concurrently. They respond with results asynchronously in any order.

PR Notes:

  • The DSL arg is solo_async=True because async is a python keyword.
  • I'm sort of lukewarm on the idea of ListenableFutures - open to suggestions. The only real benefit is that it makes testing concurrent execution/cancellation easier.
  • In theory the Sync worker pool can be cleaned up with the new interface (less of a need for a BurrowedWorkerProcess & lifecycle). I did not do this to keep this PR manageable.
Assignee
Assign to
Reviewers
Request review from
Time tracking
Source branch: github/fork/mikekap/async-worker-process