hi all,
Rust-core is getting more and more stable and is now part of recurring development
releases for Desktop, Android and iOS. Our overall release is tracked in
this github project: https://github.com/orgs/deltachat/projects/15
Dev releases are generally published to the respective github-releases pages.
https://github.com/deltachat/deltachat-desktop/releaseshttps://github.com/deltachat/deltachat-android/releaseshttps://github.com/deltachat/deltachat-ios/releases
and for iOS they also go to Testflight from time to time.
If you can help with testing releases out, filing bug issues/PRs and
checking if things are fixed ... that's much appreciated, also by
the many people who you thus help to avoid encountering the error ;)
below you find a note on the cleanup efforts after Rust-core started
existing from a C2Rust translated code base. We've come a looong way!
cheers,
holger
Rust-core cleanup effort status
-------------------------------
We got rid a few days ago of the last "OK_TO_CONTINUE" patterns
which came from C-level "goto". So that column is gone and we now have
"unsafe fn" which indicate that a whole function is marked as unsafe.
We try to push this down to the actual function parts that are unsafe.
In any case, the current stats now look like this:
dc_tools.rs unsafe: 29 free: 10 unsafe-fn: 11 chars: 100
wrapmime.rs unsafe: 49 free: 1 unsafe-fn: 2 chars: 12
dc_mimeparser.rs unsafe: 22 free: 5 unsafe-fn: 10 chars: 27
dc_receive_imf.rs unsafe: 13 free: 3 unsafe-fn: 13 chars: 12
e2ee.rs unsafe: 27 free: 6 unsafe-fn: 0 chars: 5
pgp.rs unsafe: 1 free: 0 unsafe-fn: 1 chars: 19
imex.rs unsafe: 7 free: 3 unsafe-fn: 0 chars: 3
aheader.rs unsafe: 8 free: 0 unsafe-fn: 0 chars: 3
mimefactory.rs unsafe: 5 free: 2 unsafe-fn: 1 chars: 3
error.rs unsafe: 0 free: 0 unsafe-fn: 0 chars: 5
dc_strencode.rs unsafe: 1 free: 2 unsafe-fn: 0 chars: 2
stock.rs unsafe: 2 free: 1 unsafe-fn: 0 chars: 1
job.rs unsafe: 3 free: 0 unsafe-fn: 0 chars: 0
message.rs unsafe: 1 free: 2 unsafe-fn: 0 chars: 0
configure/mod.rs unsafe: 1 free: 0 unsafe-fn: 1 chars: 0
imap.rs unsafe: 1 free: 0 unsafe-fn: 0 chars: 0
context.rs unsafe: 1 free: 0 unsafe-fn: 0 chars: 0
lib.rs unsafe: 1 free: 0 unsafe-fn: 0 chars: 0
key.rs unsafe: 1 free: 0 unsafe-fn: 0 chars: 0
total unsafe: 173
total free: 35
total unsafe-fn: 39
total c_chars: 192
The incoming-message (mime-parsing and processing) pipeline remains the last
bigger worry. We have overall 4.83% unsafe out of 20538 overall code lines in
rust-core, according to cargo-count.
Hello,
could you please explain me what the "On-demand location streaming"
feature does in Delta Chat? I need to translate this string but it's not
clear to me what it does.
Thanks a lot,
Enrico
Hi,
does anyone operate/know about an smtp/imap server that uses
a self-signed certificate? We'd like to test current rust-core
with it (it should work but is reported to not work) ...
cheers,
holger
Hi
I am using the last beta on my iPhone 6s using a Cuban nauta account. It works now!
Just a trouble, the app doesn’t erase the email after it got it.
Hi all,
below a mail i just sent to OpenTechFund internal mailing lists about what's going
on with DC -- thought it might also be interseting for folks here :)
It's not complete i notice now -- for example we also had sysadmin-related
sessions going on at UXDC regarding evolving the https://testrun.org setup
that we used for testing purposes. And also new nice graphics from janka
eg about the positioning of Delta regarding "the cloud":
https://github.com/deltachat/playground/tree/master/graphics
We are currently working in "fix-and-release" crunch mode and work on
related issues here https://github.com/orgs/deltachat/projects/15
where you can help if you are not coding, for example:
- PRs to update the web page are welcome ... we are using using jekyll
which generates a static site. It'd be great go have a team for the web
site (including FAQ) that continually improves/updates it, especially
with all the nice upcoming releases :)
- engage in the support forum https://support.delta.chat to discuss
with and help each other. Also subscribing to github-repositories
and skiming what's happening there also helps to bridge the gap
between support-forum and github in both directions. Much appreciated!
Github repos are here:
https://github.com/deltachat
- test out new desktop/ios releases and provide precise bug reports
and test fixes on github
Please note: if you start to contribute please don't start off with
suggesting we do everything differently :) Better get a feel first how
and why things happen, who is doing things, and then, over time, evolve to
conspire with other active folks (coders and non-coders) to move things
together in directions you'd like to see. Issue trackers on github are used for
a) bug reporting
b) aforementioned conspiracies on moving things forward
there are typically no "c) nice-to-haves" type-issues. Those might get
shut-down pretty fast. We want github issues to reflect what is currently
worked on, on active folk's minds, and try to keep number of issues
currently around 50.
cheers,
holger
----- Forwarded message from holger krekel <holger(a)merlinux.eu> -----
Date: Mon, 30 Sep 2019 13:45:04 +0200
From: holger krekel <holger(a)merlinux.eu>
To: otf-active(a)opentechfund.org, otf-talk(a)opentechfund.org
Subject: Delta Chat Sept 2019: Gmail/OAuth back, evolving rust-core/releases and new video
hello otf-active, otf-talk,
so some quick update highlights from Delta Chat for September:
- After producing three videos for Google and a lot of mails ...
We finally have Gmail/OAuth back!
This means that with the playstore/f-droid released Delta/Android
version you can type in your gmail address, and then you get guided
through Google's OAUTH process to grant access to your e-mail messages.
Note that Delta Chat is technically an e-mail client, it uses SMTP and IMAP
and transmits nothing to other servers.
- Our "usability-deltachat" (UXDC) gathering yielded enjoyable
sessions on preparing real-world tests, conducting a new needfinding round,
testing out and designing new multi-device workflows, and we also on the final
day produced a first video that we plan to publish once we get good new releases
out on all platforms. Here it is:
Intro-DeltaChat Video: https://www.youtube.com/watch?v=yPEjYpE_kvc
- Our Rust-core ist evolving fast, with 152 merged PRs and 4 pending
in septmeber. If you consider that Rust is a low-level systems programming
language, that's pretty impressive to be able to move like this without
getting crashes left and right. All apps/platforms use or will use soon
Rust-core for releases. New feature development takes place there and
C-development will be closed down. There remain a couple of bugs in
rust-core we need to fix but it's already well usable and used
day-to-day by several folks in the community.
- There are first desktop preview releases for Windows/Mac/Linux with a
*much* improved UX and new design. It's now much more
like Android and things are improving/evolving fast there as well. To install,
click Assets on
https://github.com/deltachat/deltachat-desktop/releases
- Heavy work on a new *much* revamped Delta/iOS version ... for those
who like numbers it's 64 merged PRs in September so far. To indicate the
level of seriousness in this effort: Bjoern who started Delta/Android
30 months ago, has switched his complete laptop/phone environment to
Apple to make sure Delta/iOS meets common expectations. There are
overall now at least a dozen very active committers.
- A new "provider-db" community effort was started to better guide users on
their initial setups -- what steps are required or if a provider is simply
expected to work. See here for a preview -- we are still refining
things and want to improve the data basis before switching this life
and bringing it into apps:
https://providers.delta.chat/
best,
holger
----- End forwarded message -----
some updates for those who are not closely following
github or IRC where most development is communicated ...
iOS and Desktop UX efforts are heavily and very nicely progressing ...
there are theming, localization, refined group-creation and chat views,
and much more, along with severe speedups and cleanups on all fronts.
More on this when we get to release them ... still a bit off but
hopefully not too much ;)
This core library porting work (126 merged PRs in the last months,
despite several vacationings and CCC camp intervening) has been done
by a growing team of existing and new developers, see here
https://github.com/deltachat/deltachat-core-rust/pulse/monthly
As to our super-scientific memory-management "evilness" metrics,
here is what happened between Jul 26, Aug 13 and today (Sept 16):
total unsafe: 567 -> 514 -> 287
total free: 326 -> 289 -> 186
total gotoblocks: 225 -> 147 -> 0
LOCs src/*.rs: 31296 -> 31150 -> 26970
The biggest source of manual memory management is now mime generation
and mime parsing. It is also often the cause for some bugs and issues
especially with incoming messages.
While the sudden development of porting the core library fro C to Rust
wholesale (started April 26th in Kyiv) has somewhat reduced our UI release frequency
it's going to be a big win in the slightly longer run. There are many new
features and improvements we want to tackle and our new Rust-core development
situation provides an excellent basis for this.
Nevertheless, Delta Chat remains a very usability focused project and
there were equally many efforts spent in improving and fixing UX issues
even though they didn't become visisble in releases yet. Next week
several of the UX-focused developers, researchers, activists and
designers are to hang out together in Freiburg and draw up plans for the
next half year of UX feature and streamlining happenings. But first
of all we'll focus on getting releases out, hopefully in September!
If you will, the new Rust-core work is also aiming at better usability, namely
for developers as they can now change more things with more confidence even if
they you don't have years of low-level programming experience. Special
thanks again here go to Friedel who carefully spear-headed and guided the porting work,
and to Alexander/link2xt and Kaction/Dimitry who joined these efforts (and the overall
DC team) six weeks ago and have made many significant contributions without which
we wouldn't be near where we are now.
Meanwhile, Google's new OAUTH verification policies drove away quite some DC users --
it's never nice if an app that you used suddenly stops working :(
The "fun" thing is that the OAUTH dev verficiation process seems to be geared
at "platform" apps that store user data on their own servers. However,
when you use DC with gmail then the only server used for storing user
data is Gmail and your end-device. There are no other servers involved
and we manage zero/zilch/nada user-data. This at least should enable us to avoid them
requiring between $15K-70K USD for a special security review as to how our (non-existing)
platform handles user-data. However, we have been bot-informed a few days ago that
we now need to produce youtube-videos showing and explaining OAUTH authentication
workings and this is what we are doing next (also quite some work).
Never mind, that Delta Chat worked nicely and caused 0 issues with
Gmail/OAUTH between Feb and July 2019 and never mind that DC comes with
a crystal-clear fully GDPR-compliant privacy policy, and its source code
is fully visible. Oh well, yandex OAUTH DC integration still works fine ;)
holger
Hi all,
some of you might have seen bjoern's post regarding the current problematics with
GMail/OAUTH: https://support.delta.chat/t/workaround-for-gmail-oauth2-simplified-setup-p…
below i'll inline a post on the Thunderbird-mailing list which is also
discussing Gmail/IMAP policy trajectories. Some folks claimed on that list that
Gmail would be actively discouraging mail apps from using imap and instead
use Gmail-special APIs. There are rumors but no announcements/links.
FYI, since beginning August DC saw a constant flow of people complaining
on twitter, DC contact addresses and other channels, about DC not working
with Gmail anymore. We had submitted a DC application (end july) to Google to be allowed
to continue to use OAUTH (it worked since Feb 2019) but despite multiple contact
attempts we have not heart a human reply, and got an initial "rejected" without
any reasons. Bjoern even tried to phone a german Google phone number but only
found a machine voice speaking some very generic 0800 numbers to try next.
Apart from taking a lot of time, this situations already damaged early adoption of
Delta Chat so it's more than unfortunate that Google's policy-makers apparently take
Kafka's Castle as an inspiration for designing social interactions.
cheers
holger
----- Forwarded message from Ben Bucksch <ben.bucksch(a)beonex.com> -----
Date: Fri, 30 Aug 2019 16:44:47 +0200
From: Ben Bucksch <ben.bucksch(a)beonex.com>
To: maildev(a)lists.thunderbird.net
Subject: Re: [Maildev] Google stopping IMAP and SMTP access an moving to proprietary protocols?
Joshua Cranmer 🐧 wrote on 30.08.19 01:26:
On 8/29/2019 3:50 PM, Ben Bucksch wrote:
Concrete things that I'm missing in our case:
* Definition of the specific SASL scheme, so that I can connect IMAP
with OAUTH2. (As said, this must work 100% identical for all
ISPs.)
... Every protocol that implements SASL is required to list the
supported mechanisms, and OAUTHBEARER is the correct SASL scheme.
Right. Just that Google, who we're talking about here, doesn't use it.
1 CAPABILITY:
Google imap.google.com:
* CAPABILITY IMAP4rev1 UNSELECT LITERAL+ IDLE NAMESPACE QUOTA ID XLIST CHILDREN X-GM-EXT-1
https://developers.google.com/gmail/imap/imap-extensionshttps://developers.google.com/gmail/imap/imap-smtp
Note the complete absense of any SASL mechanisms, not even AUTH=PLAIN or
the old AUTH=XOAUTH2. Which underlines my statement that Google doesn't
implement the standards properly, or the standards are lacking. In any
case, it's not discoverable for a generic client. You cannot support
Google without special hacks for Google. That's not a standard.
Yahoo imap.mail.yahoo.com:
* OK [CAPABILITY IMAP4rev1 SASL-IR AUTH=PLAIN AUTH=XOAUTH2
AUTH=OAUTHBEARER ID MOVE NAMESPACE XYMHIGHESTMODSEQ UIDPLUS LITERAL+
CHILDREN X-MSG-EXT OBJECTID] IMAP4rev1 Hello
They do it properly.
web.de imap.web.de:
* CAPABILITY IMAP4rev1 CHILDREN ENABLE ID IDLE LIST-EXTENDED LIST-STATUS
LITERAL- MOVE NAMESPACE QUOTA SASL-IR SORT SPECIAL-USE
THREAD=ORDEREDSUBJECT UIDPLUS UNSELECT WITHIN AUTH=LOGIN AUTH=PLAIN
They are the best, because they require OAuth2 and support all interesting
and latest extensions, like LIST-EXTENDED (instead of XLIST).
* Knowing how I get the JSON result, given that I'm in an
interactive browser session.
* Google says I should use the system web browser for their OAUTH2
login. I can launch a URL there, but wouldn't know how to follow
which URL we're at or get the JSON result from that, given that
there is no defined API that works cross-browser. Obviously, we'd
need an API for all of Windows, Mac, Linux, Android and iOS.
* The "scope" is currently completely up to the ISP to define, yet I
need to pass the right scope in. I'm dead already with this little
detail. This would need to be defined in a standard.
* Google, Microsoft and others want me to register my application
and authenticate the calls from my app. That poses 3 concrete
problems:
* ... It's practically impossible to get the secrets from all
ISPs in the world, because there are thousands. Note that all
app authors would need to get the secrets for all ISPs.
Imagine I just want to make an open-source "mail check" app.
That's practically impossible....
There are mechanisms to register the client ID dynamically, and
mechanisms for the authentication process to tell you where to find the
OAuth parameters to do the request (including endpoint locations).
However, these mechanisms are optional. We should have made
implementation in Thunderbird contingent on providers actually
supporting these flows as an incentive to get them to do the work.
If Google requires OAuth2 and doesn't support these mechanisms (could you
provide links, please?), that's not helping us, is it?
As long as I need to have special code or hardcoded values for Google,
they are not using standards. They support standards, if I can simply code
against IETF specs, without additional information for Google, and it
works.
For almost all ISPs in the world, it works without special cases. Google
and Microsoft are the big exceptions. Apple works just fine out of the box
without any special code whatsoever.
_______________________________________________
Maildev mailing list
Maildev(a)lists.thunderbird.net
http://lists.thunderbird.net/mailman/listinfo/maildev_lists.thunderbird.net
----- End forwarded message -----
tons of good changes to rust-core in the last month, 129 merged PRs :)
https://github.com/deltachat/deltachat-core-rust/pulse/monthly
thanks in particular to friedel, link2xt, kaction, jikstra, treefit!
however, Delta/iOS and Delta/Desktop already depend on rust-core and
can not do new releases because of bugs/crashes in rust-core.
I hope that in the next 2-3 weeks we get to a stable state and can
do releases again.
a few notes on this:
The python liveconfig tests are prepared to run during CI/PRs and
would catch a lot of UI-level issues. What is preventing that is
https://github.com/deltachat/deltachat-core-rust/issues/331https://github.com/deltachat/deltachat-core-rust/issues/326
Meanwhile please run liveconfig tests before offering a PR, see README for how to:
https://github.com/deltachat/deltachat-core-rust/tree/master/python#install…
If you need test accounts to set through "DCC_PY_LIVECONFIG" please mail me.
Note that if you run pytest with the "-s" option you'll see the logs/events and
where a test fails -- typically during termination which is not really
part of the test but rather of its teardown.
in general, to achieve "releasability" we need to file and look at all issues
marked as bug: https://github.com/deltachat/deltachat-core-rust/labels/bug
holger