Codegen v2 proposition

Created by: sturcotte06

OpenAPI generator v2

Motivation

This project is awesome. I've used it to generate clients for unreal engine and unity and it works very well. However, I've hit many issues during development, mostly NullPointerException occurring from partially defined OpenAPI specifications and codegen models being mostly null. The inheritance model also hinder code re-use, with common methods only available for all codegens of a given language.

The main problem I've seen is that none of the objects used to generate code are closed; they cannot be extended unless PRs are submitted to the official project. This forces people to fork the project to have their own modifications, which is detrimental to the evolution of the code base.

Prior to using openapi-generator, I had created a C# project to generate clients, and I had created a fully dynamic api that could be adapated to any code generations, from REST services to AMQP messages. To achieve this level of genericity, I had a tagging system where a specification was only a graph (sometimes with circular dependencies) and taggers would traverse the graph and add tags (i.e. string tuple) to the specification objects. This allowed to develop small reusable tagger objects that could be composed in a pipeline to generate the metadata required for the SDK generation.

One issue I had is that string tags are not very flexible, and if you put an object value instead of a string, your whole code base is filled with type checks to prevent an invalid cast. Fast-forward a few months, I had to make modifications to the unreal client we were generating, and a lot of the code was unreadable string manipulation where it was not clear whether I had a schema type, a partially resolved type, a full c++ type and what not, and debugging was a bit of a nightmare. It's not easy to change the one template per api/model, in the sense that, types cannot be collected together to be outputted in one template. A bit of magic needs to be done (i.e. supporting file magic) for everything to work out.

Proposition

This pull request proposes a v2 (not yet completed) of the codegen API. I've re-used my tagger idea, but the tags are not a string tuple anymore but a type-safe construct that disallows storing an unexpected value type. All codegen model objects are taggable and they're organized in a graph-like structure (i.e. with one parent and one-to-many children). This graph-like structure can be visited recursively, and taggers visit those codegen objects to add tags to them. For instance, there is tagger that adds template files to the relevant objects, and the templating engine now set output tags on the objects, which can be read by an output processor to output the content somewhere, on disk or in the console. Here is a description of the most important types:

• CodegenTag
  Immutable type that stores a name and a value class. Most instances are static and are used as type tokens, meant to be used as a key for maps.
• CodegenTaggable
  Interface that defines all methods to work with tags. It allows to get and set CodegenTag in a type-safe way. Most of its methods are default and use the getTags() to resolve tags.
• CodegenObject
  Abstract class that is the base class for all objects that are part of the codegen graph. All CodegenObject are CodegenTaggable, they also have a getParent() and getChildren() methods to traverse the graph and a getId() method to retrieve its unique name.
• CodegenObjectVisitor
  Functional interface to traverse a CodegenObject graph. CodegenObjectVisitor.Of<T> can be used to traverse the graph, but only visit CodegenObject of the given T type.
• CodegenTagger
  Interface that sets CodegenTag on a given CodegenObject.
• AbstractCodegenTagger
  Abstract CodegenTagger that invokes an overload of the tag() for a given CodegenObject type. Does nothing if no such method exists.
• CodegenOutputProcessor
  Interface for an object that consumes output tags to write a CodegenObject contents.
• CodegenTemplateProcessor
  Interface for an object that consumes template tags to process templates for the given CodegenObject. Generate output tags with the template's content.
• Codegen
  Interface for an object responsible of generating all code for the given OpenAPI specification.
• AbstractCodegen Abstract class for specific Codegen configurations. Subclasses mostly setup their tagging pipeline through the getTaggers() method. Two additional virtual methods onPreGenerate() and onPostGenerate() can also be overriden to customize the codegen.

These are the most important types, but the proposition is a bit more complete. Currently, the v2 prototype supports mustache (MustacheCodegenTemplateEngine) and outputs both to file and console (FileCodegenOutputProcessor and ConsoleCodegenOutputProcessor). A custom codegen can be created as such:

public class MyCodegen extends AbstractCodegen {
    public MyCodegen(CodegenTemplateProcessor templateProcessor, CodegenOutputProcessor outputProcessor) {
        super(templateProcessor, outputProcessor);
    }

    @Override
    protected Collection<CodegenTagger> getTaggers(CodegenOptions options) {
        return new ArrayList<CodegenTagger>() {{
            // Adds one template to objects of type CodegenApi
            // Adds one template to objects of type CodegenModel
            add(new CodegenTemplateTagger.Builder()
                    .withApiTemplate("myclient/api.mustache", ".java")
                    .withModelTemplate("myclient/model.mustache", ".java")
                    .build());
            // Normalizes names, getters and setters to camel case
            add(new StringNormalizerCodegenTagger(
                    StringCaseUtils::camelCase, 
                    CodegenCodeTags.NAME, CodegenCodeTags.ACCESSOR,
                    CodegenCodeTags.MUTATOR));
            // Normalizes all type names to upper camel case
            add(new StringNormalizerCodegenTagger(
                    StringCaseUtils::upperCamelCase, 
                    CodegenCodeTags.TYPE, CodegenCodeTags.BASE_TYPE, 
                    CodegenCodeTags.RETURN_TYPE, CodegenCodeTags.GENERIC_TYPES));
            // Normalizes enum values to upper snake case
            add(new StringNormalizerCodegenTagger(
                    StringCaseUtils::upperSnakeCase, CodegenCodeTags.ENUMS));
            // Add any additional taggers you may have
            // add(new MyCustomCodegenTagger());
        }};
    }
    
    @Override
    protected void onPreGenerate(OpenAPI openAPI, CodegenOptions options) {
        // Add any custom processing to do before generation
    }

    @Override
    protected void onPostGenerate(OpenAPI openAPI, CodegenSdk sdk, CodegenOptions options) {
        // Add any custom processing to do after generation
    }

    public static void main(String­[] args) {
         // [...] Retrieved from somewhere
        OpenAPI openAPI = null;

        // Create codegen from default template processor
        // that outputs generated content to System.out
        MyCodegen codegen = new MyCodegen(
                new DefaultCodegenTemplateProcessor(),
                new ConsoleCodegenOutputProcessor());
        
        // Builds codegen graph, tags all object, process
        // templates and outputs all content
        codegen.generate(openAPI, new DefaultCodegenOptions());
    }
}