[delta-chat] The state of building deltachat

Hi all, On IRC I was told the current build situation is unclear (no surprise!) and an attempt to sketch out the current and desired next state would be useful. Here's my attempt, it's only partial obviously and I probably missed many things, got many wrong and many will have changed within a few days (e.g. we're about to add a Rust dependency so who knows how that'll look). Anyway, currently deltachat-core ships with a bunch of (outdated and limited) dependencies bundled in the libs directory. Other than netpgp you should try to avoid to need these when building. However they are used for the android build which is why they can't be simply removed. I'll come back to fixing the android situation later. If you continue on unix it's probably because you want a deltachat-desktop install. The recommended way to install on linux is to use the flatpak: https://flathub.org/apps/details/chat.delta.desktop it works correctly on all linuxes right now. I'll continue the journey into how we produce said flatpak, which will bear many similarities with how to build on other unixes. Next you need to build the deltachat-node bindings. This currently builds libdeltachat.a itself rather than using the system installed version. Likewise it uses the bundled libetpan. The next improvement to make here is to change the build to use a system-installed libdeltachat.so and thus giving control to packagers to decide which dependencies are used. It means that downstream users no longer depend on the git subproject of deltachat-core which happens to be in deltachat-node, thus taking deltachat-node out of release work (unless it needs code changes itself). This is also a desired step to support packaging on linux distros who will otherwise never package deltachat-node. As a result of this we also want to stop posting prebuilt binaries of deltachat-node to npm. This will break some CIs in deltachat-desktop which will have to start building their own dependencies. Further possible scope creep here is whether to also support several variations of building, e.g. against a static libdeltachat.a? Against a non-installed version? But it seems this is not needed initially due to the slow rate of changes in deltachat-node. Next up is deltachat-desktop, like deltachat-core this does see a lot of changes for each release. This simply depends on deltachat-node, so once dc-node is updated to depend on the system libdeltachat.so everyone building this will need to build and install libdeltachat.so first. Finally the actual packaging work happens, e.g. in the flatpak repo. These need to build all dependencies, dc-core and then npm install dc-dekstop (which will pull in dc-node via npm and build it against system dc-core). This currently does not install dc-core or all of it's dependencies because dc-node currently uses a weird mix. Now to get back to Android. This currently uses all the bundled dependencies in the dc-core libs directory, it does however not use the meson build definitions instead lists all the .c and .h files in it's own build system. These bundled dependencies still are too outdated and hard to upgrade though. There are two choices here (well, maybe more - tell me!): - Do the same as on other unixes and build all the dependencies externally, pointing the android build to the ready .a files. We'd probably want to do this on a CI system. We can create a docker image which builds all these dependencies and can build dc-core & dc-android using them. - Still bundle the dependencies but make it easier to upgrade them by splitting them into their own repos, tying them in via meson subprojects. This would then also allow to cross-build each project using meson and producing the necessary .a files which then again get used in the android build. The advantage of this is that these subprojects would also give us identical builds on all unixes if we wanted. Hope this is somewhat helpful. Feel free to comment. Cheers, Floris