18 Mar
2019
18 Mar
'19
7:54 p.m.
Hi all,
On IRC I was told the current build situation is unclear (no surprise!)
and an attempt to sketch out the current and desired next state would be
useful. Here's my attempt, it's only partial obviously and I probably
missed many things, got many wrong and many will have changed within a
few days (e.g. we're about to add a Rust dependency so who knows how
that'll look).
Anyway, currently deltachat-core ships with a bunch of (outdated and
limited) dependencies bundled in the libs directory. Other than netpgp
you should try to avoid to need these when building. However they are
used for the android build which is why they can't be simply removed.
I'll come back to fixing the android situation later.
If you continue on unix it's probably because you want a
deltachat-desktop install. The recommended way to install on linux is
to use the flatpak: https://flathub.org/apps/details/chat.delta.desktop
it works correctly on all linuxes right now. I'll continue the journey
into how we produce said flatpak, which will bear many similarities with
how to build on other unixes.
Next you need to build the deltachat-node bindings. This currently
builds libdeltachat.a itself rather than using the system installed
version. Likewise it uses the bundled libetpan. The next improvement
to make here is to change the build to use a system-installed
libdeltachat.so and thus giving control to packagers to decide which
dependencies are used. It means that downstream users no longer depend
on the git subproject of deltachat-core which happens to be in
deltachat-node, thus taking deltachat-node out of release work (unless
it needs code changes itself). This is also a desired step to support
packaging on linux distros who will otherwise never package
deltachat-node. As a result of this we also want to stop posting
prebuilt binaries of deltachat-node to npm. This will break some CIs in
deltachat-desktop which will have to start building their own
dependencies.
Further possible scope creep here is whether to also support several
variations of building, e.g. against a static libdeltachat.a? Against a
non-installed version? But it seems this is not needed initially due to
the slow rate of changes in deltachat-node.
Next up is deltachat-desktop, like deltachat-core this does see a lot of
changes for each release. This simply depends on deltachat-node, so
once dc-node is updated to depend on the system libdeltachat.so everyone
building this will need to build and install libdeltachat.so first.
Finally the actual packaging work happens, e.g. in the flatpak repo.
These need to build all dependencies, dc-core and then npm install
dc-dekstop (which will pull in dc-node via npm and build it against
system dc-core). This currently does not install dc-core or all of it's
dependencies because dc-node currently uses a weird mix.
Now to get back to Android. This currently uses all the bundled
dependencies in the dc-core libs directory, it does however not use the
meson build definitions instead lists all the .c and .h files in it's
own build system. These bundled dependencies still are too outdated and
hard to upgrade though.
There are two choices here (well, maybe more - tell me!):
- Do the same as on other unixes and build all the dependencies
externally, pointing the android build to the ready .a files. We'd
probably want to do this on a CI system. We can create a docker image
which builds all these dependencies and can build dc-core & dc-android
using them.
- Still bundle the dependencies but make it easier to upgrade them by
splitting them into their own repos, tying them in via meson
subprojects. This would then also allow to cross-build each project
using meson and producing the necessary .a files which then again get
used in the android build. The advantage of this is that these
subprojects would also give us identical builds on all unixes if we
wanted.
Hope this is somewhat helpful. Feel free to comment.
Cheers,
Floris
19 Mar
19 Mar
2:43 a.m.
New subject: [delta-chat] Re: The state of building deltachat
Hey Floris,
first, thanks a lot for your build & packaging efforts!
One thing to keep in mind about system-installed
deltachat or deltachat-node is that we need to be able
to link deltachat-node against a specific core version.
For example for the gps position thing we want to link
against a specific core branch. This should be somehow
possible to make development easier. Currently all we need
is a npm install --build-from-source and deltachat-core gets
compiled, from whatever branch it's getting pulled. Also we
can't really push a new version if -node is not shipped in the
correct version. So debundling this only makes sense if
deltachat is in any distribution repositiories, which currently
is only libdeltachat on debian as far as i know.
Also it might be true that -node doesn't have a high commit
frequency, but -core has and that's the more important thing.
I don't know if we can update -core and the same built -node
bindings still work, even if the used method headers didn't
change.
For the meson things and android:
I like splitting netpgp... into own repositories. Pulling them in
is quite easy with mesons built in wrapdb. We could even add
it to https://wrapdb.mesonbuild.com/ so other people can also
easily pull those dependencies in.
So tl;dr: I'm fine with -node pointing to a system installed
libdeltachat by default but i think it makes sense to also have
a fallback variant to get it from github & compile it to make
it easier to switch between branches which use different -node
or -core versions.
~Jikstra
On 18.03.19 20:54, Floris Bruynooghe wrote:
Hi all,
On IRC I was told the current build situation is unclear (no surprise!)
and an attempt to sketch out the current and desired next state would be
useful. Here's my attempt, it's only partial obviously and I probably
missed many things, got many wrong and many will have changed within a
few days (e.g. we're about to add a Rust dependency so who knows how
that'll look).
Anyway, currently deltachat-core ships with a bunch of (outdated and
limited) dependencies bundled in the libs directory. Other than netpgp
you should try to avoid to need these when building. However they are
used for the android build which is why they can't be simply removed.
I'll come back to fixing the android situation later.
If you continue on unix it's probably because you want a
deltachat-desktop install. The recommended way to install on linux is
to use the flatpak: https://flathub.org/apps/details/chat.delta.desktop
it works correctly on all linuxes right now. I'll continue the journey
into how we produce said flatpak, which will bear many similarities with
how to build on other unixes.
Next you need to build the deltachat-node bindings. This currently
builds libdeltachat.a itself rather than using the system installed
version. Likewise it uses the bundled libetpan. The next improvement
to make here is to change the build to use a system-installed
libdeltachat.so and thus giving control to packagers to decide which
dependencies are used. It means that downstream users no longer depend
on the git subproject of deltachat-core which happens to be in
deltachat-node, thus taking deltachat-node out of release work (unless
it needs code changes itself). This is also a desired step to support
packaging on linux distros who will otherwise never package
deltachat-node. As a result of this we also want to stop posting
prebuilt binaries of deltachat-node to npm. This will break some CIs in
deltachat-desktop which will have to start building their own
dependencies.
Further possible scope creep here is whether to also support several
variations of building, e.g. against a static libdeltachat.a? Against a
non-installed version? But it seems this is not needed initially due to
the slow rate of changes in deltachat-node.
Next up is deltachat-desktop, like deltachat-core this does see a lot of
changes for each release. This simply depends on deltachat-node, so
once dc-node is updated to depend on the system libdeltachat.so everyone
building this will need to build and install libdeltachat.so first.
Finally the actual packaging work happens, e.g. in the flatpak repo.
These need to build all dependencies, dc-core and then npm install
dc-dekstop (which will pull in dc-node via npm and build it against
system dc-core). This currently does not install dc-core or all of it's
dependencies because dc-node currently uses a weird mix.
Now to get back to Android. This currently uses all the bundled
dependencies in the dc-core libs directory, it does however not use the
meson build definitions instead lists all the .c and .h files in it's
own build system. These bundled dependencies still are too outdated and
hard to upgrade though.
There are two choices here (well, maybe more - tell me!):
- Do the same as on other unixes and build all the dependencies
externally, pointing the android build to the ready .a files. We'd
probably want to do this on a CI system. We can create a docker image
which builds all these dependencies and can build dc-core & dc-android
using them.
- Still bundle the dependencies but make it easier to upgrade them by
splitting them into their own repos, tying them in via meson
subprojects. This would then also allow to cross-build each project
using meson and producing the necessary .a files which then again get
used in the android build. The advantage of this is that these
subprojects would also give us identical builds on all unixes if we
wanted.
Hope this is somewhat helpful. Feel free to comment.
Cheers,
Floris
_______________________________________________
delta mailing list -- delta(a)codespeak.net
To unsubscribe send an email to delta-leave(a)codespeak.net
8:10 p.m.
New subject: [delta-chat] Re: The state of building deltachat
On Tue 19 Mar 2019 at 03:43 +0100, Jikstra wrote:
One thing to keep in mind about system-installed
deltachat or deltachat-node is that we need to be able
to link deltachat-node against a specific core version.
libdeltachat should export it's version in it's .h file so that a
consumer of it can fail to build if it's too old.
For the meson things and android:
I like splitting netpgp... into own repositories. Pulling them in
is quite easy with mesons built in wrapdb. We could even add
it to https://wrapdb.mesonbuild.com/ so other people can also
easily pull those dependencies in.
Yes, we an move netpgp into it's own repository and even go the full
wrapdb way. It's a nice to have but not high on the priority to unblock
things. But if someone wants to work on this I'll be happy to give
pointers where I can.
So tl;dr: I'm fine with -node pointing to a system
installed
libdeltachat by default but i think it makes sense to also have
a fallback variant to get it from github & compile it to make
it easier to switch between branches which use different -node
or -core versions.
Sure, when I first created the issue on github to compile dc-node
against system dc-core I was worried about this workflow for dc-node
devs but was told it's not a high priority. But if it is, like you
argue here, let's work out a way to support it.
8:41 a.m.
New subject: [delta-chat] Re: The state of building deltachat
Hi Floris, all,
thanks for the writeup ... I'd like to look at the "state of building
deltachat" from the outcome perspective. We have at least 5 different
user-visible downloads that we want to provide:
- Android (currently works with the caveats also discussed below)
installable via f-droid, gplay and as direct apk from github releases etc.
- iOS (currently works but build is done in a ios/mac specific way)
installable via testflight
- Linux (currently Desktop works with flathub, which requires several manual
steps at least on ubuntu -- what's keeping appimage's from working btw?)
- OSX (works ASFAIK but is not signed so leads to warnings)
- Windows (does not work, some experimental deltachat-core building works)
It seems the biggest challenge remains a Windows download and having a
mostly-automated, improved way to produce the Android/iOS/Linux/OSX outcomes
that are already working.
One advantage of having dependencies inlined is that we know
which deps/versions are contained in eg "Deltachat Desktop Linux 0.200",
even if they are outdated. Inlined deps also simplify it for devs to
get started with doing PRs.
However, coming from a debian or system packager perspective, i can totally
see how inlined dependencies are considered evil ;)
But I am afraid that if we prioritize doing things "right" from the
sys-packaging perspective, the build process and steps get more complex,
and it might further delay getting the outcomes we want for users.
Also deps are indeed changing as we are starting to replace C with
Rust dependencies so in a couple of months the dependency tree
might look quite different -- is it really worth then to modularize
all the current C dependencies and make the build of
deltachat-core more flexible?
holger
On Mon, Mar 18, 2019 at 20:54 +0100, Floris Bruynooghe wrote:
> Hi all,
>
> On IRC I was told the current build situation is unclear (no surprise!)
> and an attempt to sketch out the current and desired next state would be
> useful. Here's my attempt, it's only partial obviously and I probably
> missed many things, got many wrong and many will have changed within a
> few days (e.g. we're about to add a Rust dependency so who knows how
> that'll look).
>
>
> Anyway, currently deltachat-core ships with a bunch of (outdated and
> limited) dependencies bundled in the libs directory. Other than netpgp
> you should try to avoid to need these when building. However they are
> used for the android build which is why they can't be simply removed.
> I'll come back to fixing the android situation later.
>
> If you continue on unix it's probably because you want a
> deltachat-desktop install. The recommended way to install on linux is
> to use the flatpak: https://flathub.org/apps/details/chat.delta.desktop
> it works correctly on all linuxes right now. I'll continue the journey
> into how we produce said flatpak, which will bear many similarities with
> how to build on other unixes.
>
> Next you need to build the deltachat-node bindings. This currently
> builds libdeltachat.a itself rather than using the system installed
> version. Likewise it uses the bundled libetpan. The next improvement
> to make here is to change the build to use a system-installed
> libdeltachat.so and thus giving control to packagers to decide which
> dependencies are used. It means that downstream users no longer depend
> on the git subproject of deltachat-core which happens to be in
> deltachat-node, thus taking deltachat-node out of release work (unless
> it needs code changes itself). This is also a desired step to support
> packaging on linux distros who will otherwise never package
> deltachat-node. As a result of this we also want to stop posting
> prebuilt binaries of deltachat-node to npm. This will break some CIs in
> deltachat-desktop which will have to start building their own
> dependencies.
>
> Further possible scope creep here is whether to also support several
> variations of building, e.g. against a static libdeltachat.a? Against a
> non-installed version? But it seems this is not needed initially due to
> the slow rate of changes in deltachat-node.
>
> Next up is deltachat-desktop, like deltachat-core this does see a lot of
> changes for each release. This simply depends on deltachat-node, so
> once dc-node is updated to depend on the system libdeltachat.so everyone
> building this will need to build and install libdeltachat.so first.
>
> Finally the actual packaging work happens, e.g. in the flatpak repo.
> These need to build all dependencies, dc-core and then npm install
> dc-dekstop (which will pull in dc-node via npm and build it against
> system dc-core). This currently does not install dc-core or all of it's
> dependencies because dc-node currently uses a weird mix.
>
>
> Now to get back to Android. This currently uses all the bundled
> dependencies in the dc-core libs directory, it does however not use the
> meson build definitions instead lists all the .c and .h files in it's
> own build system. These bundled dependencies still are too outdated and
> hard to upgrade though.
>
> There are two choices here (well, maybe more - tell me!):
>
> - Do the same as on other unixes and build all the dependencies
> externally, pointing the android build to the ready .a files. We'd
> probably want to do this on a CI system. We can create a docker image
> which builds all these dependencies and can build dc-core & dc-android
> using them.
>
> - Still bundle the dependencies but make it easier to upgrade them by
> splitting them into their own repos, tying them in via meson
> subprojects. This would then also allow to cross-build each project
> using meson and producing the necessary .a files which then again get
> used in the android build. The advantage of this is that these
> subprojects would also give us identical builds on all unixes if we
> wanted.
>
>
> Hope this is somewhat helpful. Feel free to comment.
>
> Cheers,
> Floris
> _______________________________________________
> delta mailing list -- delta(a)codespeak.net
> To unsubscribe send an email to delta-leave(a)codespeak.net
8:38 p.m.
New subject: [delta-chat] Re: The state of building deltachat
On Tue 19 Mar 2019 at 09:41 +0100, holger krekel wrote:
thanks for the writeup ... I'd like to look at
the "state of building
deltachat" from the outcome perspective.
Absolutely, I tried to write up what I think is the path with the
minimal amount of work to achieve a subset of the goals: android and
linux releases while not consciously sabotaging the other goals.
We have at least 5 different
user-visible downloads that we want to provide:
- Android (currently works with the caveats also discussed below)
installable via f-droid, gplay and as direct apk from github releases etc.
- iOS (currently works but build is done in a ios/mac specific way)
installable via testflight
- Linux (currently Desktop works with flathub, which requires several manual
steps at least on ubuntu -- what's keeping appimage's from working btw?)
Inside an appimage you have to craft your binary in such a way it relies
only on a very strict subset of system services. The tldr is there is
no tooling support for this, not even from appimage itself, and it is a
lot of hard work to get it right and very hard to notice you got it
wrong. OTOH flatpak does provide good tooling and guarantees that it
will work correctly, it works well with the existing ecosystems, so it's
just a much smaller and easier amount of work.
- OSX (works ASFAIK but is not signed so leads to
warnings)
- Windows (does not work, some experimental deltachat-core building works)
It seems the biggest challenge remains a Windows download and having a
mostly-automated, improved way to produce the Android/iOS/Linux/OSX outcomes
that are already working.
sure
One advantage of having dependencies inlined is that
we know which
deps/versions are contained in eg "Deltachat Desktop Linux 0.200",
even if they are outdated. Inlined deps also simplify it for devs to
get started with doing PRs.
Bundled dependencies surely could have these advantages. They're also a
lot more work to get right and come with a much higher maintenance
burden. We've wanted this for rather a long time now and no one has had
the time to step up and do all the required work. So I've been trying
to think of how to get to reliable, repeatable builds with the least
amount of work. For the long term I still think this is a fine idea to
pursue and will happily provide help if someone want to work on it.
(this scenario is basically covered by the last bullet point i wrote
about under android btw)
However, coming from a debian or system packager
perspective, i can totally
see how inlined dependencies are considered evil ;)
This wasn't a specific goal of my writeup. It just turns out that given
how the ecosystems have evolved over the last few decades that's the
easiest way to do things. (I have to add that's not what I thought I'd
be writing when I started the writeup. Again it's what I -currently-
think is the least amount of work to get to a reliable, repeatable
build)
But I am afraid that if we prioritize doing things
"right" from the
sys-packaging perspective, the build process and steps get more complex,
and it might further delay getting the outcomes we want for users.
Also deps are indeed changing as we are starting to replace C with
Rust dependencies so in a couple of months the dependency tree
might look quite different -- is it really worth then to modularize
all the current C dependencies and make the build of
deltachat-core more flexible?
I hope by now I've managed to convey that's a misunderstanding. Mind
you, I can still be mistaken too and part of the reason to write this
down was to see if people spotted ways to get to our goals with less
work.
Cheers,
Floris
20 Mar
20 Mar
11:58 a.m.
New subject: [delta-chat] Re: The state of building deltachat
Of course stripped-down renditions of the conditions are packaged with Delta Chat Core and
these will be utilized when a reliance is absent. http://www.getessaydone.com/
12:01 p.m.
New subject: [delta-chat] Re: The state of building deltachat
i unsubscribed this user -- just had admitted the subscription but
it's likely a spam-bot ... wonder if we could report this account
to gmail somehow?
holger
On Wed, Mar 20, 2019 at 11:58 -0000, Sarah Hamilton wrote:
> Of course stripped-down renditions of the conditions are packaged with Delta Chat
Core and these will be utilized when a reliance is absent. http://www.getessaydone.com/
> _______________________________________________
> delta mailing list -- delta(a)codespeak.net
> To unsubscribe send an email to delta-leave(a)codespeak.net
2103
days inactive
2105
days old
6 comments
4 participants
participants (4)
-
Floris Bruynooghe -
holger krekel -
Jikstra -
Sarah Hamilton