I'm curious about the stdlib -- given that there's no dead code removal (as far as I know), and the OCaml stdlib is basically ever-growing -- should there be really more and more code in there?

Please excuse me if the goal is not to have a small footprint of a hello-world application? Or if this is the wrong PR to ask such a question.

In my experience, using a hello world as an example (a single OCaml module using print_endline "hello") - compiled with ocamlopt to an executable, there's with 4.14 a 1.1 MB big binary (stripped 313KB), with 5.3 it is 1.6MB (stripped 394KB).

I for one would appreciate to have a plan for link-time removal of code before growing the stdlib further.

The Standard Library doesn't grow as a result of this PR (module aliases).

For your hello world example on my system, the size in bytecode increases by 45 bytes (from 22760 to 22715) - that is probably down to the addition of module Dynlink in stdlib.ml (literally the space for the "Stdlib__Dynlink" constant and a pointer or two).

In native code, there's an increase of 560 bytes, but that's more likely down to the changes in the runtime (in particular the addition of a primitive in callback.n.o) - the assembler produced is identical and stdlib.o doesn't change in size.

@dra27 thanks for your investigation and comment.

The original question and subsequent answer is interesting -- but it still leaves some questions unanswered. So if there is no dead code elimination during link time what happens generally to all the unused functions in an OCaml executable-- do they accumulate in the executable ?

So if there is no dead code elimination during link time what happens generally to all the unused functions in an OCaml executable-- do they accumulate in the executable ?

The compiler does not currently perform dead code elimination at the function level. However, it does perform dead code elimination of whole modules: modules of libraries which are completely unused (except perhaps by appearing in module aliases declarations), are removed from the final executable (unless -linkall is being used).

Everything else goes into the final executable.

Given that OCaml does not jump into the middle of functions, would it not be a not too hard task to simply remove all the functions that are not being used ? A bit surpised that this is not happening already. Is it because of dynamic linking ? The ocaml program might dynamically load a module that might might call arbitrary functions within the executable ?

Does dead code elimination happen in flambda ?

[I just realized that this discussion is probably off topic by now. Would appreciate any answer. Will take discussion on discuss.ocaml.org subsequently if it feels particularly compelling to continue it]

[I just realized that this discussion is probably off topic by now. Would appreciate any answer. Will take discussion on discuss.ocaml.org subsequently if it feels particularly compelling to continue it]

You will be interested by #608.

Given that OCaml does not jump into the middle of functions, would it not be a not too hard task to simply remove all the functions that are not being used ? A bit surpised that this is not happening already. Is it because of dynamic linking ? The ocaml program might dynamically load a module that might might call arbitrary functions within the executable ?

All functions exposed by a module are stored in the module's static block. So unless you run some kind of escaping analysis on the whole program, everything that is exposed in the signature of a module that is used has to be present in the executable.

Does dead code elimination happen in flambda ?

In a given module, yes, flambda will remove unused functions. But everything that is exported stays.
You can see #608 for an attempt to do whole program dead code elimination with flambda.

Coming back to this PR, without having gone through the details, it seems like a good simplification from a distance.

@dra27 what's the story for backwards comatibility? Are we going to ship a dummy META file so that build system dependencies on dynlink continue to work?

One specific aspect of the Dynlink library is that linking with it adds --export-dynamic on the linker's command line (by building dynlink.cmxa with something like ocamlopt -ccopt -Wl,-E). The effect is that if you link with ocamlopt -ccopt -Wl,--no-export-dynamic, then the resulting exe includes no dynamic symbol table [1], thus saving space, except for the rare executables that do use dynlink.

Doesn't this change force the dynamic symbol table to always be generated?

(That being said, while I see code related to that behavior being deleted, I don't see -Wl,-E in dynlink.cmxa even at the base of the pull request. Not sure if that's an issue with the configure script or what)

[1] not to be confused with the symbol table. IIRC, the dynamic symbol table is used by dlsym to lookup symbols (so it's probably a map from symbol to address), whereas the symbol table is used by gdb/perf to display names instead of addresses (so it's probably a map from addresses to symbols).

@v-gb - I'd wondered this a bit too, and as far as I could tell (certainly looking at 4.14) the flags are equivalent in each case - i.e. -Wl,-E is being passed irrespective of whether dynlink.cmxa is being linked. I'm guessing (without having tried) that if one has a program at present which doesn't use natdynlink (and are passing -Wl,--no-export-dynamic) that this would be unaffected by the change here. In fact, I think one would still expect a program which does use natdynlink to link even with --no-export-dynamic, I think you'd then simply see errors on the first Dynlink.loadfile, right?

Indeed, there are two levels for compatibility - we can ship a dummy dynlink.cma+dynlink.cmxa (and the old META) file or we can ship just a dummy META file. Or, has here at present, we can do nothing!

My instinct is to do neither of these things since, as we found with OCaml 5.0, the effect of putting them in place is that nothing ever plans for their removal! It's very similar to what happened with bigarray in OCaml 5.0. In Dune, users should be able to specify dynlink in their libraries stanza and have Dune determine whether that needs passing on to the linker (this is exactly the story started in ocaml/dune#5494). ocamlfind packages describe what is actually in the current compiler - i.e. for one single version of OCaml (and in OCaml 5.4 there isn't a Dynlink library, if this gets included, just as there isn't a Bigarray library and there isn't a Bytes library, etc.). For Dune it is different, because although the atoms in a libraries stanza look like findlib package names, they're semantically more complicated than that because libraries covers multiple OCaml versions (i.e. bigarray in a libraries stanza means "I expect the Bigarray module to be available, which means doing nothing for OCaml 5.0+ and pulling in the findlib bigarray package otherwise").

All that said, this would put an upper bound on all currently released versions of Dune (and any packages not using Dune, whose build systems would also need updating), which might be an argument for doing both - i.e. updating Dune et al to understand that in OCaml 5.4+ dynlink doesn't need to be processed but also allowing the META file to be shipped. If dynlink.cma and dynlink.cmxa are removed, it might also be worth having a "hint" in the driver so that ocamlopt -o foo -I +dynlink dynlink.cmxa produces a better hint than "dynlink.cmxa not found", etc.

I don't mind exactly what we do, but I do (strongly) think we should have a plan which definitely involves there not being a dynlink package/library within a few versions (as opposed to what we've done before where we say we're going to do that and then find we can't because nothing's changed in the ecosystem!). I'm happy to spearhead some of the bulk testing needed to decide on which plan for this but, unlike some of my previous efforts (#11198, #11199, #11200, etc.), I'd prefer to be certain that we're actually going to make the core change (Dynlink into the stdlib) before investing the effort!

I just looked at 4.14, and the flags are consistent with what I described above. They are, in order:

• -Wl,-E early on (no idea why, as that seems unnecessary)
• --no-export-symbols, if you use ocamlopt -ccopt -Wl,--no-export-symbols
• a second -Wl,-E, if you use dynlink

The -Wl,-E from dynlink is needed so that programs that do use dynlink execute correctly (when using ocamlopt -ccopt -Wl,--no-export-symbols).

So I have the impression that this behavior doesn't work starting in some unknown ocaml 5 version, given the apparent absence of -Wl,-E flags from dynlink.cmxa in current trunk. But if the absence of -Wl,-E was fixed (I'm guessing there's a $foo missing in a Makefile somewhere), then what I wrote originally should apply: this pull request would prevent dropping the dynamic symbol table, thus forcing larger executables, right?

In Dune, users should be able to specify dynlink in their libraries stanza and have Dune determine whether that needs passing on to the linker (this is exactly the story started in ocaml/dune#5494).

Maybe dune likes to hard code ad-hoc build logics for the libraries distributed by the compiler but there are other build systems which do not. As such I would rather prefer if the deprecation route here follows what is usually done with identifier deprecation. That would mean:

1. On release 5.4, ship empty META files that declare the dynlink library name as usual.
2. At release 5.4 + n add a warning directive to the meta file (example) that indicates that it's no longer needed to specify dynlink and that the library name will be removed in release 5.4 + n + m.
3. At release 5.4 + n + m stop distributing the META file.

Just to check I understand this correctly - at 5.4 + n + m you would therefore expect to be completely dropping support for 5.3 and earlier? In which case, what are your reasonable-sounding values for n and m?

Incidentally, rather than having to be hard-coded in the build system (which is how Dune does it, indeed), this feels more like the configure system:

checking whether the C compiler accepts -fdebug-prefix-map=old=new... yes
checking whether the OCaml compiler includes Dynlink in the Standard Library... yes

or, perhaps as a better simile:

checking for ld option to reload object files... -r
check for the ocamlopt option to use the Dynlink module... none needed

The -Wl,-E from dynlink is needed so that programs that do use dynlink execute correctly (when using ocamlopt -ccopt -Wl,--no-export-symbols).

For my understanding, what's the scenario which means that -ccopt -Wl,--no-export-symbols is being passed when it shouldn't be? In other words, rather than relying on dynlink.cmxa passing -Wl,-E to contradict this, why is not possible not to pass the -Wl,--no-export-symbols in the first place?

For my understanding, what's the scenario which means that -ccopt -Wl,--no-export-symbols is being passed when it shouldn't be? In other words, rather than relying on dynlink.cmxa passing -Wl,-E to contradict this, why is not possible not to pass the -Wl,--no-export-symbols in the first place?

The scenario is: the build system is systematically passing -ccopt -Wl,--no-export-symbols on every single link command. There is only one way to do things differently, and it's to never pass the option, ie make all executables that don't use dynlink larger. That's because the condition where -Wl,-E is necessary is known to compiler, but can't be computed by the build system (short of duplicating the linking logic of the compiler, obviously).

but can't be computed by the build system (short of duplicating the linking logic of the compiler, obviously).

The build system knew it was passing dynlink.cmxa, though? Doesn't that mean that it does know which executables are using Dynlink (and therefore shouldn't have that flag be applied)?

The build system knew it was passing dynlink.cmxa, though? Doesn't that mean that it does know which executables are using Dynlink (and therefore shouldn't have that flag be applied)?

The build system knows that dynlink.cmxa is on the command line, but that doesn't necessarily imply it's going to be kept. The difference is presumably small in this case, however if dynlink is part of stdlib, the difference between "dynlink is provided on the command line" and "dynlink is kept in the final executable" is ~100% of executables in one case vs ~0% of executables in other case.

Just to check I understand this correctly - at 5.4 + n + m you would therefore expect to be completely dropping support for 5.3 and earlier?

Or leave it to build/configure systems to handle the conditional churn if they want to.

In which case, what are your reasonable-sounding values for n and m?

That's for upstream to determine. But six years ago @xavierleroy asked me to write a deprecation procedure for OCaml, the thoughts are still here, including baselines for timings¹.

@xavierleroy I wonder if you have an opinion and would be willing to "shepherd" the PR -- make a general decision and maybe ask someone to review it. (I thought about @nojb maybe.)

I can do a review of the code changes, if there is agreement on the general direction.

I haven't looked at the code, just read @dra27's excellent description. Overall, I think that sharing more code between the Dynlink library and the toplevel REPLs is a good thing, but I'm not convinced Dynlink belongs to the standard library. For me, the standard library is not "whatever is needed by the compilers", it's "basic functionality that we encourage everyone to use and rely on". Dynlink is a fairly niche functionality, it's hard to use (owing to OCaml's strict link-time consistency checking), and its API is clumsy, so I would not encourage everyone to use it and rely on it.

@dra27:

However, having moved Dynlink into the Standard Library, this opens the possibility of using it for the toplevel. That's presently not possible because ocaml is part of the core system, and the core system cannot use otherlibraries1. There's nothing particularly magic about Dynlink, in terms of the physical OCaml code which it consists of - unlike the other libraries in otherlibs, Dynlink's C code already lives entirely in the runtime; it's just the dependencies which were a problem.

Could we consider the opposite solution then, of moving the toplevel outside the "core"? I don't see any obvious downside to that.

@v-gb:

The build system knows that dynlink.cmxa is on the command line, but that doesn't necessarily imply it's going to be kept.

This is confusing me - if dynlink.cmxa is on the command line, it will be kept, or at least I'm 99% certain its C flags will be, which is what matter here. So if a build system is passing dynlink.cmxa on the command line, it would already in 5.2.0 and earlier I think be losing the effect of any -ccopt -Wl,--no-export-symbols on the same command line.

Can I ask whether this is a hypothetical build you're referring to, or an actual one, because if it's a real one I'd find it much easier to be looking at it. If it's not a real one, I'm veering towards being less convinced that the change here would be an insurmountable problem.

Totally independently of that, but if #13285 has caused flags not to be in dynlink.cmxa which used to be, then that should at least be triaged and possibly fixed (but I haven't had a chance to look at that).

@dra27:

Or, has here at present, we can do nothing!

I'm not sure what I was smoking¹. We should obviously not be breaking existing code at the instant of the change. My personal intuition (as with the various attempts I've made at explaining future directions for stdlib-shims) is that the deprecation warning should appear immediately (that doesn't break released code, because of warnings are never treated as errors in releases, right 😉). So I would probably push for n=0 (I don't immediately see the benefit to delaying the advert that something needs changing, but that may be that I just don't see it) and that m would be relatively small, possibly reviewed at each release in the light of the results of bulk package tests.

Thanks for the review of the first part, @nojb - I'm certainly happy to split the subsequent parts off (as a small scheduling matter, I'm away all next week, so delayed replies then will at least be intentional, for a change)!

I have mused before if we actually want something more akin to runtime.cmxa (Marshal, Obj, various bits of Sys, all the Camlinternal modules etc.) and stdlib.cmxa (the actual "Standard Library" stuff), so the idea that Dynlink doesn't strictly belong in the Standard Library resonates.

I'm not so sure about removing the toplevel from the boot cycle, @gasche. I'm not sure why, but that doesn't quite feel right to me. However, we could go in a slightly different direction, perhaps, and instead promote dynlink.cma out of otherlibs and into the bootstrap (which is what this PR is already doing, of course) but retain it as dynlink.cma. In theory that simplifies various things - dynlink.cma and dynlink.cmxa are retained (and with them any C flags which should or should be there) and there would clearly still be a META file for a dynlink package. However, it would then be trivial for the bytecode toplevel to be linked against dynlink.cma in the same way that that native toplevel is linked against dynlink.cmxa. That would affect things which tried to link against ocamltoplevel.cma, but I would bet on there not being (m)any things which link against ocamltoplevel.cma and don't either do that via the META file (i.e. they wouldn't notice the change, because dynlink.cma would be specified for them).

I can knock that up as an experimental branch, assuming it's not an instantly repulsive idea to anyone?

The build system knows that dynlink.cmxa is on the command line, but that doesn't necessarily imply it's going to be kept.

This is confusing me - if dynlink.cmxa is on the command line, it will be kept, or at least I'm 99% certain its C flags will be, which is what matter here. So if a build system is passing dynlink.cmxa on the command line, it would already in 5.2.0 and earlier I think be losing the effect of any -ccopt -Wl,--no-export-symbols on the same command line.

Linking works this way (ignoring -linkall, as it is orthogonal to this discussion):

• .cmx files given at the command line are linked in
• any .cmx file contained in command line .cmxa are linked in if they are referred to from a linked-in compilation unit

So dynlink.cmxa at the command line doesn't have to be linked in. I agree the flags will be included no matter what, which is strange. But I don't know why we're discussing this.

My point is merely that: there are flags attached to dynlink.cmxa (although that got broken somewhere in the 5.x versions), and they should be attached to dynlink.cmx here. The effect of the flags can be mostly emulated by the build system before this pull request, but not after.

Can I ask whether this is a hypothetical build you're referring to, or an actual one, because if it's a real one I'd find it much easier to be looking at it. If it's not a real one, I'm veering towards being less convinced that the change here would be an insurmountable problem.

I'm talking about an actual build. I don't have access to the build anymore, not that there'd be anything interesting to see. Just imagine https://github.com/ocaml/dune/blob/d826e9b80284794979d57508d80d3d76143d96de/src/dune_rules/exe.ml#L210-L234 listing -ccopt -Wl,-E as the first option on the compiler's command line.
There's clearly nothing unsurmountable here, it'd be enough to attach -Wl,-E flags to dynlink.cmx instead of dynlink.cmxa, and problem solved. But, going from memory, flags cannot currently be attached to .cmx files, so it requires modest ocamlopt-level changes.