Implementing the Tuple API easily using custom annotations and reflections

Hey, so I've been working on something interesting lately. It's a solution to a problem that you may have encountered in one form or anothing but it boils down to this: you have a struct:

struct mystruct
{
    long a;
    int b;
    char c;
};

And you want this to work:

auto [a, b, c] = instance;

Well it doesn't, aggregates don't implement the tuple API required for structured bindings by default and implementing it manually is a pain. You need to write your own specialization of std::tuple\size, std::tuple_element and implement a visible get<> method. It might not be that much work at first glance but it becomes very tiresome if you have multiple such structs or if you try to deviate from the general default indexing and not to mention a paint to maintain and update.

Well, why write repeated code when you can only write code once that writes that code for you? Reflections are mature enough in gcc's implementation to solve this. Simply put I have compiling code that with only the application of an annotation gives me all of this:

struct [=tuple_like] simpletuple
{
    long a;
    int b;
    char c;
};

Through which I get:

staticassert(std::tuplesizev<simpletuple> == 3);


constexpr simpletuple simple { 1, 2, 3 };


staticassert(1 == get<0>(simple));
staticassert(2 == get<1>(simple));
staticassert(3 == get<2>(simple));


staticassert(std::sameas<std::tupleelementt<0, simpletuple>, long>);
staticassert(std::sameas<std::tupleelementt<1, simpletuple>, int>);
staticassert(std::sameas<std::tupleelementt<2, simpletuple>, char>);

And of couse:

auto a, b, c = simple;

It doesn't stop here though, since the case I described above is common but inflexible. Customization is available and an opt-in feature:

struct [=tuple_like] customizabletuple
{
    constexpr customizabletuple(long a, const char b, float c, char d, double e)
        : a(a)
        , b(b)
        , c(c)
        , d(d)
        , e(e)
    {}


    [[=tuple_element(0)]]
    long a;
    [[=read_only_tuple_element(3)]]
    const char b;
    [=tuple_element(2)]
    float c;
    char d;


private:
    [=read_only_tuple_element(1)]
    double e;
};

It supports:

tuple element index reordering
implicit data member exclusion from the API
explicit read-only modifier
explicit opt-in for private fields
correct semantics for members which are reference types

&#8203;

constexpr customizable_tuple custom( 1, "2", 3, 4, 5 );

static_assert(std::tuple_size_v<customizable_tuple> == 4);

static_assert(1 == get<0>(custom));
static_assert(std::string_view{"2"} == get<3>(custom));
static_assert(3 == get<2>(custom));
static_assert(5 == get<1>(custom));

static_assert(std::same_as<std::tuple_element_t<0, customizable_tuple>, long>);
static_assert(std::same_as<std::tuple_element_t<1, customizable_tuple>, const double>);
static_assert(std::same_as<std::tuple_element_t<2, customizable_tuple>, float>);
static_assert(std::same_as<std::tuple_element_t<3, customizable_tuple>, const char const>);

But it doesn't stop there, no it doesn't. The part that took the most effort to implement were the error messages. No, we have better static_assert's in C++26 now, we abuse them:

Example 1:

staticassert(3 == get<3>(simple));

/app/myreflectutils.hpp:819:17:

error:
static assertion failed: reflectutils: get<3> called for tuple-like type 'simpletuple', but its reflected tuple size is 3.

Example 2:

[[=tupleelement(1)]]
long a;
[=tuple_element(2)]
int b;
[=tuple_element(3)]
char c;

/app/myreflectutils.hpp:632:21: error: static assertion failed: reflectutils: invalid tuple layout for 'simpletuple': tuple index 0 is missing. There are 3

explicitly annotated members, so their indices must be exactly 0, 3). Member 'simple_tuple::c' uses index 3.

Example 3:

[[=tuple_element(0)]
[=tuple_element(1)]
long a;

/app/myreflectutils.hpp:632:21: error: static assertion failed: reflectutils: invalid tuple layout for 'simpletuple': member 'simpletuple::a' has 2 tuple annotations; use exactly one of [[=tupleelement(i)]] and [=read_only_tuple_element(i)].

Full code: https://godbolt.org/z/exjW6YGEr

https://redd.it/1v9u2d5
@r_cpp

C++26: Reducing undefined behaviour
https://www.sandordargo.com/blog/2026/07/29/cpp26-reduces-undefined-behaviour

https://redd.it/1v9x7nt
@r_cpp

Building a compiler that works at compile-time so you can compile your program while you compile your program.

In short, I wanted to build a compiler of some C subset that would work at compile-time. It compiles into a custom byte-code for a runtime VM.

I've once tried to write a compile-time C compiler, but I abandoned that project, because I made it overly complex (one-pass compiler right into x86). No clear separation between parser, lexer, etc.

Why would I even want this? Idk. But how can it be useful?
- The code of the compiler doesn't go to the resulting binary,
- No need to waste time for compilation at runtime too,
- Guaranteed type-safety. There can't be such thing as "oh, I changed the function signature, but forgot to update the bindings and it crashed at runtime"

And I shouldn't forget about cons:
- No optimisations. Real compilers spent decades on them and I'm definitely not going to implement LLVM at runtime. Although we could make a compile-time x86 VM, so we can run it at compile time... no, thank you, it's a topic for another fever dream article.
- Hot-reload! I mean, no hot-reload. I won't even mention it anymore, considering that the script is compiled at compile-time and is builtin right into the binary file. I could implement it with hot memory patching or smth, but who really needs it.

Let's start.
## Bypassing constexpr limitations
C++ 20 lets us to dynamically allocate memory at compile-time and even use std::vector that really expands our borders. But there's one very important note - you can't declare a compile-time vector and extract it into the runtime. No constexpr std::vector<int> data = makeData();, it won't compile. So we need to hack it.

### Passing strings in templates
Sadly, the C++ Committee made a lot of cool compile-time features, but not enough (at least for me). We still can't use strings in templates without hacks. But we can easily bypass it with a well-known trick.

template<std::size_t N>  
struct const_string {  
  constexpr const_string() = default;  
  
  // implicit-constructor that lets us to do bad things
  constexpr const_string(const char (&str)[N]) {
    std::copy_n(str, N, value);  
  }  
  
  constexpr operator std::string_view() const {  
    return {value, value + N - 1};  
  }  
  
  char value[N]{};  
  const std::size_t length = N;  
};

// using it
template<const_string str>
auto very_smart_function(...) { /* ... */ }

### Extracting vectors from compile time
It turned out to be not really that hard, but I didn't really find any ready examples on Internet, unlike with const_string.
To extract std::vector<T> from constexpr we need to make it std::array<T, N> somehow. The main problem is that we can't write std::array<T, myVector.size()>, because myVector.size() won't be a constant value. So we must to make it constant somehow.
I thought of passing vector as a template parameter, but we can't do it legally. C++ 20 allows us to pass only the structs with all-public members. After deeply thinking a bit (not really), I discovered that I could simply pass the lambda that returns our vector (I didn't think I could just pass a pointer actually).

// data_getter is our lambda
template<auto data_getter>
constexpr auto to_array() {  
  using value_type = typename decltype(data_getter())::value_type;  
  constexpr static std::size_t size = data_getter().size();  
  
  // Create a static array with a "dynamic" size and copy all data
  std::array<value_type, size> out;  
  auto in = data_getter();  
  for (std::size_t i = 0; i < size; ++i) {  
    out[i] = in[i];  
  }  
  return out; // yay
}

template<const_string str>
constexpr auto lex() {
 constexpr static auto data_getter = [] constexpr {
  // .lex() returns the vector of tokens
  return lexer{static_cast<std::string_view>(str)}.lex();
 };
 // All our data are available for runtime now =D
 return to_array<data_getter>();
}

### Printing errors
For nice errors C++ has static_assert that allows us to even print our custom message! But it must be always a literal (until C++

Building a compiler that works at compile-time so you can compile your program while you compile your program.

In short, I wanted to build a compiler of some C subset that would work at compile-time. It compiles into a custom byte-code for a runtime VM.

I've once tried to write a compile-time C compiler, but I abandoned that project, because I made it overly complex (one-pass compiler right into x86). No clear separation between parser, lexer, etc.

Why would I even want this? Idk. But how can it be useful?
- The code of the compiler doesn't go to the resulting binary,
- No need to waste time for compilation at runtime too,
- Guaranteed type-safety. There can't be such thing as "oh, I changed the function signature, but forgot to update the bindings and it crashed at runtime"

And I shouldn't forget about cons:
- No optimisations. Real compilers spent decades on them and I'm definitely not going to implement LLVM at runtime. Although we could make a compile-time x86 VM, so we can run it at compile time... no, thank you, it's a topic for another fever dream article.
- Hot-reload! I mean, no hot-reload. I won't even mention it anymore, considering that the script is compiled at compile-time and is builtin right into the binary file. I could implement it with hot memory patching or smth, but who really needs it.

Let's start.
## Bypassing constexpr limitations
C++ 20 lets us to dynamically allocate memory at compile-time and even use `std::vector` that really expands our borders. But there's one very important note - you can't declare a compile-time vector and extract it into the runtime. No `constexpr std::vector<int> data = makeData();`, it won't compile. So we need to hack it.

### Passing strings in templates
Sadly, the C++ Committee made a lot of cool compile-time features, but not enough (at least for me). We still can't use strings in templates without hacks. But we can easily bypass it with a well-known trick.
```cpp
template<std::size_t N>
struct const_string {
constexpr const_string() = default;

// implicit-constructor that lets us to do bad things
constexpr const_string(const char (&str)[N]) {
std::copy_n(str, N, value);
}

constexpr operator std::string_view() const {
return {value, value + N - 1};
}

char value[N]{};
const std::size_t length = N;
};

// using it
template<const_string str>
auto very_smart_function(...) { /* ... */ }
```

### Extracting vectors from compile time
It turned out to be not really that hard, but I didn't really find any ready examples on Internet, unlike with `const_string`.
To extract `std::vector<T>` from constexpr we need to make it `std::array<T, N>` somehow. The main problem is that we can't write `std::array<T, myVector.size()>`, because `myVector.size()` won't be a constant value. So we must to make it constant somehow.
I thought of passing vector as a template parameter, but we can't do it legally. C++ 20 allows us to pass only the structs with all-public members. After deeply thinking a bit (not really), I discovered that I could simply pass the lambda that returns our vector (I didn't think I could just pass a pointer actually).

```C++
// data_getter is our lambda
template<auto data_getter>
constexpr auto to_array() {
using value_type = typename decltype(data_getter())::value_type;
constexpr static std::size_t size = data_getter().size();

// Create a static array with a "dynamic" size and copy all data
std::array<value_type, size> out;
auto in = data_getter();
for (std::size_t i = 0; i < size; ++i) {
out[i] = in[i];
}
return out; // yay
}

template<const_string str>
constexpr auto lex() {
constexpr static auto data_getter = [] constexpr {
// .lex() returns the vector of tokens
return lexer{static_cast<std::string_view>(str)}.lex();
};
// All our data are available for runtime now =D
return to_array<data_getter>();
}
```

### Printing errors
For nice errors C++ has `static_assert` that allows us to even print our custom message! But it must be always a literal (until C++

26)
```cpp
constexpr auto parse() {
// Allowed
static_assert(false, "Expected ';'");

// Not allowed :( (until C++ 26)
std::size_t line = 5;
static_assert(false, "Expected ';' at line " + to_string(line));
}
```
I didn't want my project to require C++ 26, so I used another trick. The formatted string gets turned into a static array just like a vector (into `const_string` actually) and then it's passed into `ErrorMessage<const_string Msg>` that triggers compilation error. So we force the compiler to print the full type name that includes our error. But sadly the type name has a limit about 100 symbols. I think I could solve it with splitting the message into several ErrorMessages... God, I don't want to read this in my console.
```c++
template<const_string Msg>
struct ErrorMessage {
static_assert(false, "Check the template parameter for details");
};

template<auto err_getter>
consteval auto report_error() -> void {
// C++ 26 support
#ifdef KORKA_FEATURE_FORMATTED_STATIC_ASSERT
static_assert(false, to_string(err_getter()));
#else
constexpr auto msg = const_string_from_string_view<[] { return to_string(err_getter()); }>();
std::ignore = ErrorMessage<msg>{};
#endif
}
```

I don't want you to see it, so I'll just show C++ 26 version.
```
error: static assertion failed: Lexer Error: Unterminated string at line 12
```

### Mapping signatures to names. And vice versa
In our little runtime C++ we are used to `std::unordered_map<string, value_t>` and other standard or non-standard (hello, Boost!) containers. But I needed a table where a key is a string and the value is a TYPE. And in C++ I can't treat types as values, I can't just put them into a dict... :(

So, welcome another hack!
```c++
template<auto, class>
struct signature_mapper;

// function_info_getter takes an index to our mapped function,
// and Is... holds all indices
template<auto function_info_getter, std::size_t... Is>
struct signature_mapper<
function_info_getter,
std::index_sequence<Is...>
> {
// hash func
consteval static auto hash(auto &&v) -> std::size_t {
return frozen::elsa<std::string_view>{}(v, 0);
}

// Our function overloaded with many unique types based on hash of the mapped function
constexpr static auto _overloaded = overloaded{
(
[](unique_type<hash(function_info_getter(Is).name)>)
-> const_function_info_to_signature_t<[] { return function_info_getter(Is); }> * {
return nullptr;
}
)...
};

// Extracting the type by name
template<const_string name>
using get_signature_t = std::remove_pointer_t<decltype(
_overloaded(
unique_type<hash(name)>{}
)
)>;

};
```

We use well-known function overload (~~but for evil things~~). Basically, one type inherits a lot of lambdas that take an empty `unique_type<hash>` that serves as our key and returns the pointer to our type.
```cpp
// How our mapper looks after expanding our params
struct overloaded : lambda1, lambda2, lambda3 {
using lambda1::operator();
using lambda2::operator();
using lambda3::operator();
};

// And every lambda looks like this
auto lambda_fib = [](unique_type<hash("fib")>) -> signature_of_fib* { return nullptr; };
```
When we call `_overloaded(unique_type<hash(name)>())` our poor compiler has to resolve the overload. And he looks for right one through all `()` operators. And then we just take that it returns (our `T*`) and get the `T`.
I use this "mechanism" to extract script functions into the native C++.
```cpp
constexpr auto script_fib = compile_result.function<"fib">();
```

### Bindings from C++ to our script lang
This was the most exhausting part. Well, how "exhausting" exactly... I was thinking for a few evenings and then made it work one morning. The problem was with me. I wanted to make a pretty API that was impossible in the current standard (maybe it's possible in C++ 26, but I didn't check it).
I wanted it to look like this:
```cpp
auto func() -> void;
auto foo(int) -> int;

// Примерно так
constexpr auto bindings = korka::make_bindings<
"func", func,
"foo", foo
>();

// Или так
constexpr auto

bindings = korka::make_bindings(
"func", func,
"foo", foo
);
```
But why couldn't I make it work?
In C++ you can't pass a string into `template <auto ...args>`. We need `const_string`. We can't mix types in the one stream of variadic args and make compiler guess it right. Templates require explicitness and it's impossible to write a universal parser.
Variant #2 works, but you can't extract the function into the runtime. You just can't. Functions may have different signatures, but you need to make them all the same type, and create a FFI wrapper along the way. Compile-time doesn't allow `reinterpretet_cast<void*>(&func)`.

So I designed this:
```cpp
constexpr auto bindings = korka::make_bindings(
korka::wrap<fib>("cpp_fib"),
korka::wrap<print_n>("print_n")
);
```
Not so elegant, but still not bad.
`wrap` is very simple
```cpp
// our FFI signature
using vm_external_function_type = void(vm::context_base &context);

// info for our compiler
template<class Signature>
struct wrapped_function {
using signature_t = Signature;

vm_external_function_type &external_func;
std::string_view name;
};

template<auto func>
consteval auto wrap(std::string_view name) {
return wrapped_function<std::decay_t<decltype(func)>>{
binding_wrapper<func>,
name
};
}
```
The most interesting part is inside `binding_wrapper<func>`. I won't show the full code here, because I still didn't tell about the VM architecture that will execute it. But in short binding_wrapper just checks the signature, generates some code that extracts arguments from VM, calls native functions and then puts the result back. Simple.
## The compiler and the VM
Maybe the most interesting part of the article. I have never written any compilers before (the thing I mentioned in the beginning of the article doesn't count), so I made it according to the first articles I found in Google.

Compiler has 3 modules:
- the lexer - splitting the code into tokens,
- the parser - building a tree from the tokens,
- the compiler itself - making the tree into byte-code. And doing semantic analysis at the same time (I was too lazy to make another module)
I think I could compose everything into one class via composition or smth, but it's too late already.

### Lexer
Primitive. We just look for tokens in a loop until we reach EOF.
```cpp
constexpr auto scan_token() -> std::optional<std::expected<lex_token, error_t>> {
char c = advance();
switch (c) {
case '{':
return make_token(lex_kind::kOpenBrace);
case '}':
return make_token(lex_kind::kCloseBrace);
case '(':
return make_token(lex_kind::kOpenParenthesis);
case ')':
// ...

case ' ':
case '\r':
case '\t':
// Ignore whitespace
return std::nullopt;

// ...

default:
if (is_digit(c)) {
return scan_number();
} else if (is_alpha(c)) {
return scan_identifier();
}
}
}
```

### Parser
More interesting. We need to build the AST (abstract syntax tree). And we need to store this tree somehow. The usual way with `Node` that keeps pointers to other nodes won't do, because we're at compile-time. I mean, we can write it this way, it will work, but extracting this tree into compile time? No. We would need serialisation or something. So we can use simple trick with `std::vector<Node>` and just make nodes store indices to each other.
This approach also increases the cache locality of the data for the CPU, but I doubt the CPU will be even aware of our "smart" trick, since everything is executed at compile-time.

The parser is recursive, while parsing one expression we parse another. Small fragment of the code:
```cpp
constexpr auto parse_statement() -> parse_result {
auto tok = peek();
if (!tok) return make_error("Unexpected end of input");

switch (tok->kind) {
case lex_kind::kOpenBrace: return parse_compound_stmt(); // { ... }
case lex_kind::kIf: return parse_if_statement(); // if (...) ...
case lex_kind::kWhile: return parse_while_statement(); //

while (...) ...
case lex_kind::kReturn: return parse_return_statement(); // return ...;
default: return parse_expression_stmt(); /// ...;
}
}

constexpr auto parse_return_statement() -> parse_result {
if (!match(lex_kind::kReturn)) return make_error("Expected 'return'");

index_t expr_idx = empty_node; // empty_node = -1
if (auto next = peek(); next && next->kind != lex_kind::kSemicolon) {
auto expr = parse_expression(); // another recursive call
if (!expr) return std::unexpected{expr.error()};
expr_idx = *expr;
}

if (!match(lex_kind::kSemicolon)) return make_error("Expected ';' after return");
return m_pool.add(stmt_return{expr_idx});
}
```
parse_return_statement goes into parse_expression, that goes into parse_assigment, that goes into parse_logical_or, that goes... Well, you got it. That's how operator priority works here.

### Their Majesty Compiler (and analyser)
I may have cheated here a bit.

Before we even write a compiler, we must know for what architecture we do it. x86, ARM or even JVM.
Initially when I was working on a similar project, I planned to generate raw assembly for x86 (last versions of Clang and GCC support passing `constexpr std::string_view` into `asm(...)` statement), but honestly writing a compiler for a zoo of x86 instructions is the right way to madhouse.

And even so, if we downgrade our compiler we won't have nice constexpr asm anymore. And we can't also generate raw machine instructions because of DEP (data execution prevention). We'll have to call non-crossplatform `mmap` or `VirtualAlloc` to allocate some memory, copy the code there... Good riddance cross platform build compler, hello Windows Defender that will kill our app for such tricks with memory.

So where have I cheated? I made my own architecture that will execute inside a VM. A stack VM.
Why stack it? It turned out to be incredibly easy to generate the bytecode for. If you are doing a register architecture (as in processors or Lua), then you will have to write register allocation algorithms (it is difficult). And in the stack everything is much simpler.

If we need to sum A and B we just do this:
1. Put A into the stack.
2. Put B into the stack.
3. Execute sum instruction. It takes these two values and puts back their sum.

So I had this set of instructions at the end:
```cpp
enum class op_code : char {
// Loads/saves locals to/from the stack (variables).
lload, lsave,

i64_const, // Puts a constant onto the stack

// Math
i64_add, i64_sub, i64_mul, i64_div,

// Puts 1 if values are equal (i made <, <= later)
i64_cmp,

jmp, // jumps by offset
jmpz, // conditional jump by offset, only when 0 on the stack

call, // calls a function
ret, // returns from the function

trap, // calls a native C++ function
};
```

### So what about the analysis?
In classical compilers phases are strictly splitted: lexers builds tokens, parser builds tree, semantic analyser checks the types and variables, then optimisations, then codegen and then optimisations again.

As you can remember, I'm pretty lazy. And keep in mind that `constexpr` ops are not infinite. I didn't want to make a separate pipeline phase. So my compiler combines these two functions: semantic analysis and code generation.

They usually call it Single-Pass Compilation, but it's not really the case here, since it's only about these two phases. My compiler is a bit hybrid.

My compiler recursively walks the tree and does two things:
1. Checks the semantics: "was this variable declared and what's its type" before we even try to multiply something. Do function param types match? Does this function even exist? Etc.
2. Generates the byte-code. If semantics is ok, then we just write corresponding instructions immediately into the `std::vector<std::byte>` (the one we're going to elegantly extract via `to_array`).
And in the result we receive a ready, semantically-correct and absolute safe (let's pretend that I wrote the compiler bug-free, huh) byte-code that we feed to the

VM.

## So what do we have?
Let's look how the API of my poor lib looks (let's call it Korka).

This example uses bindings (100% type safe, I swear on the standard):
```cpp

// Our native C++ functions
auto fib(std::int64_t n) -> std::int64_t {
if (n == 0) return 0;
if (n == 1) return 1;
return fib(n - 1) + fib(n - 2);
}

auto print_n(std::int64_t n) -> void {
std::cout << n << '\n';
}

// Our not-so-native script
constexpr char code[] = R"(
int fib(int n) {
if (n == 0) return 0;
if (n == 1) return 1;
return fib(n-1) + cpp_fib(n-2);
}

void print_fib(int n) {
int result = fib(n);
print_n(result);
return;
}
)";

// We create bindings
constexpr auto bindings = korka::make_bindings(
korka::wrap<fib>("cpp_fib"),
korka::wrap<print_n>("print_n")
);

// Compile at compile-time, yay
constexpr auto compile_result = korka::compile<code, &bindings>();

// Extract function adressess + their types
constexpr auto script_fib = compile_result.function<"fib">();
constexpr auto script_print_fib = compile_result.function<"print_fib">();

int main() {
// Init VM
korka::vm::context ctx{compile_result.bytes, bindings};

// Call fib that returns int64_t
auto result = ctx.call(script_fib, 12L);
std::cout << "fib(12) = " << result << '\n'; // prints 144

// Call print_fib
ctx.call(script_print_fib, 16L); // prints 987

return 0;
}
```

Ta da! It works.

## Small analysis

Out of curiosity, I decided to compare Korka with other scripting languages. A pretty API is great, sure, but was it worth the effort performance-wise? So, let's pit Korka head-to-head against Python and Lua.

For the benchmark, I used the recursive calculation of the $N$-th Fibonacci number, an excellent test to fairly evaluate overhead on function calls, stack management, and overall runtime efficiency (the first thing that came to my head).

I tested everything on a franken-server put together from spare parts, powered by an Intel Xeon E5-2689 (3.6 GHz).

I measured two stages:
- **Initialization time** from runtime startup to being ready to execute the first instruction,
- **Execution time** of the algorithm itself.
### Stage 1: Initialization

|**Language / Library**|**Initialization time**|
|---|---|
|**Korka**|**1.5 µs**|
|**Lua**|152.6 µs|
|**Python**|25,097.0 µs|

Korka takes the lead: it starts 100 times faster than Lua and over 15,000 times faster than Python.

The explanation is simple: while Lua and Python are busy reading the script at startup, parsing it, compiling it into their byte-codes, and spinning up heavy infrastructure (including the GC), Korka does not. All the virtual machine has to do is grab the pre-compiled output (and allocate a tiny bit of memory).

### Stage 2: Runtime

After a series of optimizations, the results turned out pretty solid (I know comparing statically typed and dynamic languages isn't entirely fair, but who's gonna stop me?):

| **N** | **Iterations** | **Korka (ms)** | **Lua (ms)** | **Python (ms)** | **vs. Python** |
| ------ | -------------- | -------------- | ------------ | --------------- | -------------- |
| **10** | 100 000 | **1 055,67** | 1 534,98 | 1 591,10 | **1,51x** |
| **15** | 50 000 | **5 492,40** | 8 149,17 | 8 753,77 | **1,59x** |
| **20** | 20 000 | **24 263,39** | 36 268,86 | 38 459,88 | **1,59x** |
| **23** | 10 000 | **51 367,20** | 76 494,22 | 82 257,71 | **1,60x** |
| **25** | 5 000 | **67 441,69** | 100 850,20 | 108 836,54 | **1,61x** |
| **28** | 2 000 | **114 341,21** | 169 626,06 | 184 869,57 | **1,62x** |
| **30** | 1 000 | **149 190,02** | 223 229,86 | 241 838,28 | **1,62x** |


### Conclusion
I built a (mostly) full-fledged C compiler that runs entirely in `constexpr`. Why? No idea. Especially considering it's been done before, you can check out [constexpr-8cc](https://github.com/keiichiw/constexpr-8cc). But that one lacks C++ bindings and cross-platform support.

The source code is

available on [GitHub](https://github.com/PyXiion/pxkorka) (warning: ugly code ahead!). Any feedback and comments are more than welcome.

*P.S. This article is an English translation of a post I originally published on Habr a while ago. Keep in mind that some benchmarks and discussions here may be dated.*

https://redd.it/1v9v41m
@r_cpp

const_cast: A Necessary Evil
https://www.elbeno.com/blog/?p=1858

https://redd.it/1v9zcrn
@r_cpp

Build timings from auto-modularised VS solution with external and internal libs

To investigate how well modules work with MSVC without rewriting projects and putting up with malfunctioning intellisense, I made a little python script to modularise a whole VS solution (yes, *I* made it, using real bio-neurons).

The script lets me try out different modularisation strategies on a VS Solution on a per-project basis:

* Header-based, don't modularise. If dependencies are modules, they will be imported.
* PartitionsMI: Partition-per-header using standard module cpp units.
* PartitionsPI: Partition-per-header using MSVC partition cpp units.
* Submodules: Module-per-header
* SubmodulesII: Move implementation to interface (not always possible due to cyclic imports)

Modularised code is surrounded with `extern "C++" { export {`.

The code is a super secret incomplete game engine/game. But here's some info:

* External libs, including: Boost (unordered flat map), std, d3d, imgui, physx, spdlog, <windows>.
* Wrapper module for each.
* Solution projects: CommonStuff, Scripting, GameData, Graphics, Anim, SceneGraph, SceneManager, Entity, App
* Some projects don't have much code. Others have quite a bit. Enough to give the build system something to parallelise.

All projects built using debug config. Benchmarks average 3 or more timings, outliers ignored. CPU is 12600, 12 threads. Multipliers are speedup.

| |CommonStuff|Scripting|GameData|Graphics|Anim|SceneGraph|SceneManager|Entity|
|:-|:-|:-|:-|:-|:-|:-|:-|:-|
|Header-based, no pch|4.59s|5.68s|7.46s|2.92s|40.74s|24.94s|2.86s|3.19s|
|Header-based, pch|3.61s|5.30s|6.55s||19.38s|12.53s|||
|Extlib modules + CommonStuff submodules||3.60s|4.78s|1.62s|18.55s|9.41s|1.57s|2.09s|
|PartitionsMI|4.25s||8.09s|2.54s|||||
|PartitionsPI|4.02s|6.04s|7.09s|2.53s|33.83s|15.97s|2.66s|4.10s|
|Submodules|3.59s|4.75s|6.05s|2.26s|26.80s|14.31s|2.12s|3.15s|
|SubmodulesII|3.44s||7.93s||||||

| |CommonStuff|Scripting|GameData|Graphics|Anim|SceneGraph|SceneManager|Entity|
|:-|:-|:-|:-|:-|:-|:-|:-|:-|
|Header-based, no pch|1.00x|1.00x|1.00x|1.00x|1.00x|1.00x|1.00x|1.00x|
|Header-based, pch|1.27x|1.07x|1.14x||2.10x|1.99x|||
|Extlib modules + CommonStuff submodules||1.58x|1.56x|1.80x|2.20x|2.65x|1.82x|1.53x|
|PartitionsMI|1.08x||0.92x|1.15x|||||
|PartitionsPI|1.14x|0.94x|1.05x|1.15x|1.20x|1.56x|1.08x|0.78x|
|Submodules|1.28x|1.20x|1.23x|1.29x|1.52x|1.74x|1.35x|1.01x|
|SubmodulesII|1.33x||0.94x||||||

| |App|Single file|Full Rebuild| |App|Single file|Full Rebuild|
|:-|:-|:-|:-|:-|:-|:-|:-|
|Header-based, no pch|65.19s|3.97s|145.49s||1.00x|1.00x|1.00x|
|Header-based, app pch|24.74s|2.15s|108.92s||2.64x|1.85x|1.34x|
|Header-based, mostly pch|||78.87s||||1.84x|
|Extlib modules|26.82s|2.35s|58.89s||2.43x|1.69x|2.47x|
|Extlib modules + CommonStuff partitions|25.89s|2.47s|||2.52x|1.61x||
|Extlib modules + CommonStuff submodules|24.53s|2.23s|57.50s||2.66x|1.78x|2.53x|
|Extlib modules + most libs as submodules|31.14s|2.97s|||2.09x|1.34x||
|Full submodules, except app|34.61s|2.95s|88.90s||1.88x|1.35x|1.64x|
|Full submodules|64.02s|6.22s|142.98s||1.02x|0.64x|1.02x|

* "Most libs" = CommonStuff + Scripting + GameData + Graphics + Anim + SceneGraph.
* "Mostly pch" = PCH for CommonStuff, Scripting, GameData, Anim, SceneGraph, App.
* "Single file" is recompiling one moderately complex file in App. \~930 lines, lots of includes.
* "Full Rebuild" excludes extlib module wrappers.

Detailed timings of CommonStuff:

Header-based:

1> 210 ms LIB 1 calls
1> 4459 ms CL 2 calls
04.874 seconds

PartitionsMI:

1> 125 ms LIB 1 calls
1> 199 ms MSBuild 16 calls
1> 269 ms CppClean 1 calls
1> 798 ms SetModuleDependencies 10 calls
1> 1232 ms CL 2 calls
1> 1439 ms MultiToolTask

1 calls
04.295 seconds

Submodules:

1> 141 ms MSBuild 16 calls
1> 775 ms SetModuleDependencies 10 calls
1> 1015 ms CL 2 calls
1> 1132 ms MultiToolTask 1 calls
03.427 seconds

The CL entry is for cpp files and it's 4.39x as fast as header-based. Great!

But it's the ixx dependency scanning and compilation (MultiToolTask) that brings things down. Things that aren't there with header-based.

MultiToolTask *is* parallelised though. Reducing thread count slows it down. And you can open up the msbuild binlog in the msbuild log viewer to see the parallelisation of ixx compilation. See which modules are on the end of the DAG slowing things down.

So, I don't know how MultiToolTask could be sped up, but, msbuild seems to be waiting for all ixx files before moving onto cpp files, so that's some parallelism left on the table.

Detailed timings of App:

Header-based, app pch:

1> 196 ms MSBuild 13 calls
1> 1000 ms Link 1 calls
1> 23270 ms CL 16 calls
24.708 seconds

Extlib modules + CommonStuff submodules:

1> 199 ms SetModuleDependencies 18 calls
1> 606 ms MSBuild 27 calls
1> 1197 ms Link 1 calls
1> 21877 ms CL 15 calls
24.078 seconds

Full submodules:

1> 459 ms MSBuild 27 calls
1> 1324 ms Link 1 calls
1> 4646 ms SetModuleDependencies 18 calls
1> 12593 ms MultiToolTask 1 calls
1> 49327 ms CL 15 calls
01:09.245 minutes

Slow. But looking at the log viewer, ixx compilation seems very well parallelised.

And 4.6s just to scan for dependencies. VS isn't launching a whole CL process for each file is it?

Also, why does CL take so long? All modules have been compiled by that point, so why would it be much slower?

Recompiling a single file from App:

Header-based, no pch:

1> 46 ms SetModuleDependencies 10 calls
1> 120 ms MSBuild 5 calls
1> 3757 ms CL 1 calls
04.095 seconds

Header-based, app pch:

1> 56 ms SetModuleDependencies 10 calls
1> 121 ms MSBuild 5 calls
1> 1793 ms CL 2 calls
02.059 seconds

Extlib + CommonStuff modules (as submodules):

1> 121 ms SetModuleDependencies 18 calls
1> 216 ms MSBuild 17 calls
1> 1793 ms CL 1 calls
02.162 seconds

Full submodules, except app:

1> 518 ms MSBuild 17 calls
1> 631 ms SetModuleDependencies 18 calls
1> 2211 ms CL 1 calls
03.095 seconds

Full submodules:

1> 559 ms MSBuild 17 calls
1> 696 ms SetModuleDependencies 18 calls
1> 1891 ms MultiToolTask 1 calls
1> 3221 ms CL 1 calls
06.331 seconds

That MultiToolTask is quite slow, checking that all the modules in the project are up to date it seems. Does it have to be this slow?

"CL.exe will run on 0 out of 224 file(s) in 224 batches. Startup

phase took 2040.9372ms."

Anyway.

Conclusion:

* I now have the data to know how I should go about modularising a codebase for build perf.
* Surprisingly, there is a middle-ground between 0% modules and 100% modules where you get optimal build performance. It seems extlibs and your common lib should be modularised and nothing else. Or maybe my internal lib headers are too small to benefit.
* Because of that, I might be able to have a header-based fallback for intellisense without it spreading to the rest of the code.
* Multiple modules > Partitions. Partitions have worse performance. I assume that's because they need to have an extra module at the end of the DAG.
* Their only advantage is shared module attachment across multiple ixx files, if you don't use `extern "C++"`. As I said before, I wish you could have "public" partitions or multiple modules with the same name attachment.
* MSVC partition cpp files are slightly faster vs module cpp files. But why. Cpp files wait for all ixx files to compile anyway. Noise in the data?
* MSBuild parallelises as it should, it seems, except that cpp files wait for all ixx files. Could improve performance by merging the tasks?
* cpp compilation can be much faster, but overall build perf brought down by ixx files and scanning.
* Moving implementation to ixx files doesn't always help. Need more data.
* And it would be ideal if build systems avoided the build cascade when only changing implementation details.
* The more modules you add to the solution, the more bloated the build system feels. Lots of scanning, lots of ixx checking. Can performance be improved?
* PCH can be beaten in some cases.

Problems:

* Intellisense. Red squiggles everywhere. Not only `import std` issues, but Intellisense is highly sensitive to errors in imported modules. It cannot limp along like it can with headers.
* A simple example would be missing a semicolon after a struct def in a header and in a module. Intellisense can still use the struct from the header, but not the module.
* Or in working code, something from `import std` that Intellisense can't handle.
* This means that parsing issues with Intellisense may be viral when using modules. Intellisense must either be perfect or tolerant.
* From a previous modularisation attempt, I removed the combination of explicit template instantiations of a class with constrained friend functions, due to ICE, which has now become a bogus compiler error. Bug not fixed.
* Lots of linker warnings from dllexport-ed manual RTTI data. Bug not fixed.

At least the compiler functions well enough to produce a working program with no further changes. That's pretty good.

https://redd.it/1va18lf
@r_cpp