rust-core and all the rest / rust sprinting end sept? - delta

26 Jul 2019

As some of you are aware there were a lot of changes
the last weeks in rust-core:
https://github.com/deltachat/deltachat-core-rust/pulse/monthly

Thanks to everybody who contributed and of course most to @dignifiedquire
who kick-started and guides the crazy efforts on April 26th, 2019 now
known as rust core or DCC-RS sometimes. 

However, master is currently less stable than it was beginning June -- i guess
that 4e41dbf5ab0bf23de57ea6c0841b50047b382ff0 was still quite stable FYI.

On Master there are bugs like "E2E-encryption unavailable unexpectedly" (#233)
and 
incoming messages appearing corrupted (#223). At the same time, it's not easy 
to debug those as there still is a lot of hard-to-read code that stems from 
the original C2Rust translation.  Here is an example of a merged PR that
removes some of those "goto" reminescences:
https://github.com/deltachat/deltachat-core-rust/pull/229

The idea here and probably with many other commits is to *separate functional changes 
from re-indentation changes* to allow precise review.  It's very easy to introduce
logical bugs if no one can properly compared "before/after".  Cleanup PRs
typically
should come in a Two-Commit form: one commit changing only logic (minimize indentation
changes)
and one only doing "cargo fmt".  You typically only review the first commit
and trust that the committer really did nothing else than "cargo fmt" in
the second. Cleanup PRs can be easily done by Rust-newcomers and they 
greatly help readability and debuggability of everything.  Here is
the current count of unsafe/free/goto usages in core:

    dc_tools.rs                    unsafe:  91 free:  59 goto-blocks:   4
    dc_msg.rs                      unsafe:  60 free:  38 goto-blocks:   3
    dc_chat.rs                     unsafe:  70 free:  30 goto-blocks:   0
    dc_imex.rs                     unsafe:  14 free:  30 goto-blocks:  48
    dc_configure.rs                unsafe:  15 free:  20 goto-blocks:  38
    dc_contact.rs                  unsafe:  51 free:  19 goto-blocks:   0
    dc_mimeparser.rs               unsafe:  29 free:  29 goto-blocks:  12
    dc_job.rs                      unsafe:  28 free:  11 goto-blocks:  29
    dc_receive_imf.rs              unsafe:  16 free:  22 goto-blocks:  19
    dc_securejoin.rs               unsafe:  13 free:  14 goto-blocks:  29
    ...
    ...
    total unsafe: 653
    total free: 436
    total gotoblocks: 234

Note that the latter total is down ~20% compared to June
8/4e41dbf5ab0bf23de57ea6c0841b50047b382ff0:

    total unsafe: 768
    total free: 565
    total gotoblocks: 294

The list can be obtained by executing "python top_level_evil.py" in the src
dir.

It'd be good to accelerate this cleanup-by-cleanup effort in order to help overall
stability
and readability.  You may look at the "dc_*.c" counterpart file in
https://github.com/deltachat/deltachat-core/tree/master/src
if editing a rust-file -- often there are also comments which are good to transfer. 

Floris' note made sense to me here: "it probably will get worse before it gets
better". 
As long as we have tons of unsafe/safe boundaries, changing some of them 
can easily corrupt others. 

There also are deeper cleanup-PRs like the past sqlite, strbuilder ones or the current 
https://github.com/deltachat/deltachat-core-rust/pull/211 (dc_array as Rust vector) 
https://github.com/deltachat/deltachat-core-rust/pull/188 (new stock strings API) 

Such cleanup-PRs require more Rust-knowledge already but also have a much deeper
pleasurable effect ;) I btw recommend all rust-beginners to watch the repo
and follow PRs and comments.  Lots of little things to learn there ;) 

There are some more ideas on larger scale cleanups (requiring deeper knowledge), eg:

- remove all usage of char* path-handling code contained in dc_tools.rs 
  with rust Path usage.  This also removes a lot of alloc/free code in 
  many places. 

- encasuplate (and finally) replace "mmime" (the mime-parser C2Rust translated 
  from libetpan) such that core code (dc_mimeparser as consumer and producer
  of APIs) does not leak C-pointers left and right and forces one to use 
  "carray" or "clist" in regular core-rust vectors. I gues that 
  Friedel/@dignifiedquire and Bjoern/@r10s might have more refined 
  ideas on how this could be done. 

- please tell ;) 

Lastly, i mentioned the Rust-sprint idea but i am myself not available 
in August and first half september there already is a UX/DC gathering
which focuses strongly on UX issues, so not good to mix with rust-core.

It seems likely that in the week of Sept 23-30th several folks
interested in DC and/or Rust will hang out in Freiburg/DE to do hack 
sessions and discussions around rust-core together.   Also the-compiler
and me intend to use 2 days in that week to advance DC chat bots and
maybe start a PyQT-based DC client for added fun.  If anyone wants
to join but lacks money to make the travel, please mail me.

On a related side note Bjoern and Richy are continuing to focus
on getting a major reworked "Delta/iOS" version out soon (tm). 
Delta/iOS uses rust-core so fixing the aforementioned bug-showstoppers 
remains a priority as well. Also, OAUTH2 is not working again yet (#119). 

I think i am not the only one looking very much forward to having 
rust-core become stable and nice :) It is the single basis for all DC apps and
will allow many interesting new feature and UX developments ... on which
several of us also want to engage in 2019 ... For one, there are the longer standing
issues from our Cuban friends who need better control over data/server-storage
usage ... but also needs from people who need alternatives when Signal,
Whatsapp or other centralised services are blocked ... which happens and 
threatens to happen in many places these days.  

so much for now, keep up all the good work and thanks for any help!
holger (who is mostly afk in August)

P.S.: subscription to mailing list: 
https://lists.codespeak.net/postorius/lists/delta.codespeak.net/
chatting with folks: #deltachat on freenode