Here is Excorporate version 0.6.0.
New in this release:
– Support for overriding settings autodiscovery by manually configuring an EWS URL
– A check to prevent soap-client conflicts with other packages
– Handle user@sub.domain.com/autodiscover.domain.com autodiscovery case
– Support for Exchange 2013 (via the new API addition, exco-server-version)
Thanks to everyone who tested autodiscovery on the 0.5.4 release and provided feedback. I’ve listed most of you in the Acknowledgements section.
Excorporate has been accepted for inclusion in GNU ELPA but I’m still waiting on copyright assignment for one dependency before releasing it there. This release still takes the form of a tarball. First download it, then install it with M-x package-install-file.
I would love to use excorporate 0.6.0 in our company network, but unfortunately did not succeed until now. This would allow me to completely avoid Windows for communication purposes.
I use emacs 24.4 on Linux and tried the autodiscovery process, which terminates with an error message “A TLS packet with unexpected length was received.”
I turned on debugging for the gnutls library and the output was as follows:
gnutls.c: [2] ASSERT: dn.c:1209
gnutls.c: [2] ASSERT: dn.c:1209
gnutls.c: [2] ASSERT: mpi.c:609
gnutls.c: [2] ASSERT: dn.c:1209
Contacting host: autodiscover.domain.com:443
gnutls.c: [2] (Emacs) Deallocating x509 credentials
Excorporate: Found service URL for “martin.apel@domain.com”: https://ews-eu.sub.domain.com/ews/exchange.asmx
gnutls.c: [1] (Emacs) allocating credentials
gnutls.c: [2] (Emacs) allocating x509 credentials
gnutls.c: [2] (Emacs) using default verification flags
gnutls.c: [1] (Emacs) setting the trustfile: /etc/ssl/certs/ca-certificates.crt
gnutls.c: [1] (Emacs) gnutls callbacks
gnutls.c: [1] (Emacs) gnutls_init
gnutls.c: [1] (Emacs) got non-default priority string: NORMAL
gnutls.c: [1] (Emacs) setting the priority string
gnutls.c: [2] ASSERT: gnutls_constate.c:695
gnutls.c: [2] EXT[0x89f7730]: Sending extension SAFE RENEGOTIATION (1 bytes)
gnutls.c: [2] EXT[SIGA]: sent signature algo (4.2) DSA-SHA256
gnutls.c: [2] EXT[SIGA]: sent signature algo (4.1) RSA-SHA256
gnutls.c: [2] EXT[SIGA]: sent signature algo (2.1) RSA-SHA1
gnutls.c: [2] EXT[SIGA]: sent signature algo (2.2) DSA-SHA1
gnutls.c: [2] EXT[0x89f7730]: Sending extension SIGNATURE ALGORITHMS (10 bytes)
gnutls.c: [1] (Emacs) non-fatal error: Resource temporarily unavailable, try again. [400 times]
gnutls.c: [2] ASSERT: gnutls_buffers.c:344
gnutls.c: [2] ASSERT: gnutls_buffers.c:625
gnutls.c: [2] ASSERT: gnutls_record.c:969
gnutls.c: [2] ASSERT: gnutls_handshake.c:2762
gnutls.c: [0] (Emacs) fatal error: A TLS packet with unexpected length was received.
gnutls.c: [2] errno: 32
gnutls.c: [2] ASSERT: gnutls_buffers.c:431
gnutls.c: [2] ASSERT: gnutls_buffers.c:755
gnutls.c: [2] ASSERT: gnutls_record.c:491
gnutls.el: (err=[-9] A TLS packet with unexpected length was received.) boot: (:priority NORMAL :hostname ews-eu.sub.domain.com :loglevel 2 :min-prime-bits 256 :trustfiles (/etc/ssl/certs/ca-certificates.crt) :crlfiles nil :keylist nil :verify-flags nil :verify-error nil :callbacks nil)
gnutls.c: [2] ASSERT: gnutls_buffers.c:344
gnutls.c: [2] ASSERT: gnutls_buffers.c:625
gnutls.c: [2] ASSERT: gnutls_record.c:969
gnutls.c: [2] (Emacs) Deallocating x509 credentials
Does this mean anything to you? The site uses a certificate which is signed by a custom authority, for which I have imported the certificates into the /etc/ssl/certs/ca-certificates.crt. I have played around with curl on the command line and my impression is, that there is a redirect for the wsdl page involved. Curl tells me:
> GET /ews/exchange.asmx HTTP/1.1
> Authorization: Basic
> User-Agent: curl/7.38.0
> Host: ews-eu.sub.domain.com
> Accept: */*
>
< HTTP/1.1 302 Found
< Cache-Control: private
< Content-Type: text/html; charset=utf-8
< Location: /ews/Services.wsdl
* Server Microsoft-IIS/7.5 is not blacklisted
< Server: Microsoft-IIS/7.5
< Set-Cookie: exchangecookie=3c7e90d8d21d4f57a327d5dc99b3c386; expires=Thu, 14-Apr-2016 12:15:11 GMT; path=/; HttpOnly
< X-WSSecurity-Enabled: True
< X-WSSecurity-For: None
< X-AspNet-Version: 2.0.50727
< X-Powered-By: ASP.NET
< Date: Tue, 14 Apr 2015 12:15:11 GMT
< Content-Length: 139
Is it possible, that your code does not support the redirect?
Thanks for this great idea,
Martin
Hi Martin,
Everything looks fine from the Excorporate side. The autodiscovery step works fine. I’m thinking there is something going wrong during TLS negotiations. This might represent a GnuTLS bug or misconfiguration. You can start debugging with the gnutls-cli program:
gnutls-cli ews-eu.sub.domain.com
If that doesn’t work, see if OpenSSL does work:
openssl s_client -host ews-eu.sub.domain.com -port 443
Then continue debugging from there, trying different GnuTLS versions and configurations.
The redirect does not go across domain names, so I don’t think that’s the issue.
Hi Thomas,
sorry for the late reply, I only just saw, that you already replied a few days ago. I tried the approach using gnutls-cli and everything seems to work. No TLS error messages or the like. I will try to investigate this further and keep you informed.
Thanks,
Martin
Hi Thomas. I’ve just installed this, not expecting too much of a 0.6.0 release, but it works! Autodiscovery was flawless. I am suitably impressed. Thanks for your good work. I have one comment and one question please. The comment first. My init.el now says:
(setq load-path (append (list “~/.emacs.d/excorporate-0.6.0”) load-path))
(require ‘excorporate)
(require ‘excorporate-calendar)
I’d be happy to hear of a better way to do this, or to know if I’m missing something.
Now the question. When I M-x calendar, and then e, I see the appointments for a selected day listed. [victory!] Specifically, I see the meeting time, subject, location, and invitee list. If I view the same appointments in Outlook or Thunderbird/Lighning, and expand them, I can see the detailed info. I my case, that usually has teleconferencing data such as phone numbers and meeting IDs. How to I persuade Calendar/Excorporate to display this level of detail about a meeting?
I emphasise that I don’t want this to seem like criticism. This already works better than I anticipated, and has the potential to be a real boon to my working day.
c
Hi Colin,
Thanks for the feedback.
If you installed Excorporate using M-x package-install-file, then package.el should take care of everything for you. Try just commenting out those init.el lines, restart Emacs, then M-x excorporate, M-x calendar, e should still work.
I just hacked up some prototype code to pull the body of the message. I’ll have to investigate a good UI for showing it. I’ll try to put this in the next release.
Thomas
Hi Thomas. Strangely, I don’t see excorporate when I do M-x package list. I installed by downloading the 0.6.0 archive. Really appreciate your quick feedback. Looking forward to future enhancements.
Thanks again for this! I’ve had another attempt, but I’m getting the error ‘Excorporate: Failed to retrieve https://webmail/‘. Any suggestions as to how I debug?
I can access https://webmail/owa/ with a web browser, supplying ‘DOMAIN\username’ as a username, with appropriate password. I can also access the URL with curl:
$ curl -k “https://webmail/owa/” -u “DOMAIN\username”
There is a wsdl file available, which I can access with:
curl -k “https://webmail/EWS/Services.wsdl” -u “DOMAIN\username”
I’ve tried both:
(setq excorporate-configuration (cons “username@ourdomain.com” “https://webmail/owa/”))
and
(setq excorporate-configuration (cons “username@ourdomain.com” “https://webmail/”))
but in both cases, I’m prompted for a username and password, and I get ‘Excorporate: Failed to retrieve https://webmail/owa/‘ whether I use the correct username/password or now.
I’m on Emacs 24.4 (with gnutls-verify-error set to nil) and my Outlook web access version is 8.3.379.2.
s/or now/or not/
Hi Wilfred,
The service URL you’re configuring seems wrong. Can you try:
(setq excorporate-configuration (cons “username@ourdomain.com” “https://webmail/EWS/exchange.asmx”))
Then Excorporate will try to retrieve the Services WSDL file via:
https://webmail/EWS/exchange.asmx?wsdl
You can try that in a web browser to confirm that works.
Thomas
Marvellous, I’m tantalisingly close now.
I found that the following didn’t work:
(switch-to-buffer (url-retrieve-synchronously “https://webmail/EWS/Services.wsdl”))
in Emacs 24.4, consistently giving me HTTP 401. However, in Emacs 24.5, it works.
So, I tried excoporate in Emacs 24.5. This complained about a missing exco-url-http-24.5.el, so I copied exco-url-http-24.4.el, and replaced ‘24.4’ with ‘24.5’ in that file.
I was then able to run M-x excorporate and after about 10 seconds saw ‘Excorporate: Connection (“username@ourdomain.com” . “https://webmail/EWS/Services.wsdl”) is ready’ in the *Messages* buffer. Exciting! 🙂
So, it’s looking really good. I then run M-x calendar and hit ‘e’ on a date that I know contains appointments. This opens an *Excorporate* buffer but it just shows [(==) (updating…)].
Looking at the *fsm-debug* buffer, I see https://gist.github.com/Wilfred/0bf7c9f697a71073e1ad — errors of the form ‘Error in exco–fsm/:retrieving-data: Wrong number of parameters for FindItem: expected 5, got 6’.
Any idea how I proceed from here? Is there a bug tracker or mailing list where we could exchange patches, to make your life easier?
Thanks again 🙂
The switch-to-buffer, url-retrieve-synchronously test is a good one. I would expect it to work on Emacs 24.4 though. Did you run that test before or after loading Excorporate?
As for the error you’re getting on Emacs 24.5, can you paste the value of the variable exco-server-version?
To hack it you can change exco-calfw-maybe-extra-argument in excorporate-calfw.el to always return ‘().
I’d like to understand the root causes here in case others run into these issues.
Good question about mailing list/bug tracker. Ultimately I’d like to use GNU ELPA project infrastructure, like bug-gnu-emacs, but the project isn’t in GNU ELPA yet. For now this is probably the best forum.
OK, I’ve played with this some more.
I can actually get
(switch-to-buffer (url-retrieve-synchronously “https://webmail/EWS/Services.wsdl”))
to work. However, I have to get my password correct the first time: if I screw up on the first attempt, the second password is reject unconditionally. This confused me before: I’m actually get the same behaviour on Emacs 24.4 and 24.5. Both work if I get my password right the first time.
I’m doing the following:
(require ‘excorporate)
(setq excorporate-configuration (cons “username@ourdomain.com” “https://webmail/EWS/Services.wsdl”))
and then `M-x excorporate’. This works, giving me:
Excorporate: Connection (“username@ourdomain.com” . “https://webmail/EWS/Services.wsdl”) is ready
However, `M-x calendar’ followed by `e’ still doesn’t get beyond “(==) (updating…)”. The *fsm-debug* buffer shows:
Tue Jun 16 11:53:03 2015: Starting exco–fsm
Tue Jun 16 11:53:03 2015: exco–fsm enters :retrieving-service-xml
Tue Jun 16 11:53:03 2015: Sent :retrieve-xml to exco–fsm in state :retrieving-service-xml
Tue Jun 16 11:53:12 2015: Sent :success to exco–fsm in state :retrieving-service-xml
Tue Jun 16 11:53:12 2015: exco–fsm enters :parsing-service-wsdl
Tue Jun 16 11:54:13 2015: Sent t to exco–fsm in state :parsing-service-wsdl
Tue Jun 16 11:54:14 2015: Sent t to exco–fsm in state :parsing-service-wsdl
Tue Jun 16 11:54:14 2015: exco–fsm enters :retrieving-data
Tue Jun 16 11:55:37 2015: Sent “FindItem” to exco–fsm in state :retrieving-data
Tue Jun 16 11:55:37 2015: Error in exco–fsm/:retrieving-data: Wrong number of parameters for FindItem: expected 5, got 6
I modified the function as you suggested:
(defun exco-calfw-maybe-extra-argument (server-version)
“If SERVER-VERSION is less than 2013 then return an empty list.
Otherwise return a list with one nil element.”
‘())
and restarted Emacs. This still produces the same issue:
Tue Jun 16 13:56:31 2015: Starting exco–fsm
Tue Jun 16 13:56:31 2015: exco–fsm enters :retrieving-service-xml
Tue Jun 16 13:56:31 2015: Sent :retrieve-xml to exco–fsm in state :retrieving-service-xml
Tue Jun 16 13:56:39 2015: Sent :success to exco–fsm in state :retrieving-service-xml
Tue Jun 16 13:56:39 2015: exco–fsm enters :parsing-service-wsdl
Tue Jun 16 13:58:44 2015: Sent t to exco–fsm in state :parsing-service-wsdl
Tue Jun 16 13:58:44 2015: Sent t to exco–fsm in state :parsing-service-wsdl
Tue Jun 16 13:58:44 2015: exco–fsm enters :retrieving-data
Tue Jun 16 13:58:58 2015: Sent “FindItem” to exco–fsm in state :retrieving-data
Tue Jun 16 13:58:58 2015: Error in exco–fsm/:retrieving-data: Wrong number of parameters for FindItem: expected 5, got 6
Regarding the server version:
ELISP> exco-server-version
*** Eval error *** Symbol’s value as variable is void: exco-server-version
ELISP> (exco-server-version nil)
“Exchange2007_SP1”
I’m running out of ideas, any suggestions?
I’ve tried hacking the relevant function:
(defun exco-calfw-get-meetings-for-day (month day year callback)
“Return the meetings for the specified day.
MONTH, DAY and YEAR are the meeting month, day and year. Call
CALLBACK with two arguments, IDENTIFIER and RESPONSE.”
(let* ((start-of-day-time-internal
(apply #’encode-time `(0 0 0 ,day ,month ,year)))
(start-of-day-date-time
(exco-format-date-time start-of-day-time-internal))
(start-of-next-day-date-time
(exco-extend-timezone
(format-time-string “%FT00:00:00%z”
(time-add start-of-day-time-internal
(seconds-to-time 86400))))))
(exco-operate
nil
“FindItem”
`(;; Main arguments.
((Traversal . “Shallow”)
(ItemShape
(BaseShape . “AllProperties”))
;; To aid productivity, excorporate-calfw automatically prunes your
;; meetings to a maximum of 100 per day.
(CalendarView (MaxEntriesReturned . “100”)
(StartDate . ,start-of-day-date-time)
(EndDate . ,start-of-next-day-date-time))
(ParentFolderIds
(DistinguishedFolderId (Id . “calendar”))))
;; Empty arguments.
nil nil nil nil) ;; Changed this line — Wilfred
callback)))
Which gives me this in the logs:
Tue Jun 16 14:09:53 2015: Starting exco–fsm
Tue Jun 16 14:09:53 2015: exco–fsm enters :retrieving-service-xml
Tue Jun 16 14:09:53 2015: Sent :retrieve-xml to exco–fsm in state :retrieving-service-xml
Tue Jun 16 14:09:57 2015: Sent :success to exco–fsm in state :retrieving-service-xml
Tue Jun 16 14:09:57 2015: exco–fsm enters :parsing-service-wsdl
Tue Jun 16 14:10:20 2015: Sent t to exco–fsm in state :parsing-service-wsdl
Tue Jun 16 14:10:20 2015: Sent t to exco–fsm in state :parsing-service-wsdl
Tue Jun 16 14:10:20 2015: exco–fsm enters :retrieving-data
Tue Jun 16 14:10:46 2015: Sent “FindItem” to exco–fsm in state :retrieving-data
Tue Jun 16 14:12:52 2015: Sent “FindItem” to exco–fsm in state :retrieving-data
and this in *Messages*
Contacting host: webmail:443
error in process filter: if: peculiar error: 405
error in process filter: peculiar error: 405
Mark set
Contacting host: webmail:443
error in process filter: if: peculiar error: 405
error in process filter: peculiar error: 405
OK, you’re on Exchange2007_SP1, which I don’t have available to test on. I suspect Exchange2010 added an extra argument to FindItem; if you just remove one of the nils arguments, it may just work.
In addition to the ‘exco-calfw-maybe-extra-argument’ change, can you change ‘exco-calfw-get-meetings-for-day’ to remove one of the nil values in “nil nil nil nil nil”, re-evaluate that function, then try pressing ‘e’ again in the Calendar buffer?
I tried that, see http://test.fitzsim.org/blog/?p=247#comment-32531 . The full traceback is:
signal(error (http 405))
(if error-status (signal (car error-status) (cdr error-status)) (apply callback (soap-parse-envelope (soap-parse-server-response) operation wsdl) cbargs))
(let ((error-status (plist-get status :error))) (if error-status (signal (car error-status) (cdr error-status)) (apply callback (soap-parse-envelope (soap-parse-server-response) operation wsdl) cbargs)))
(unwind-protect (let ((error-status (plist-get status :error))) (if error-status (signal (car error-status) (cdr error-status)) (apply callback (soap-parse-envelope (soap-parse-server-response) operation wsdl) cbargs))) (and (buffer-live-p data-buffer) (kill-buffer data-buffer)))
(let ((data-buffer (current-buffer))) (unwind-protect (let ((error-status (plist-get status :error))) (if error-status (signal (car error-status) (cdr error-status)) (apply callback (soap-parse-envelope (soap-parse-server-response) operation wsdl) cbargs))) (and (buffer-live-p data-buffer) (kill-buffer data-buffer))))
(closure …)
url-http-activate-callback()
url-http-wait-for-headers-change-function(1 319 318)
url-http-generic-filter(# “HTTP/1.1 405 Method Not Allowed
\nAllow: GET, HEAD, OPTIONS, TRACE
\nServer: Microsoft-IIS/7.0
\nX-Powered-By: ASP.NET
\nDate: Tue, 16 Jun 2015 15:16:20 GMT
\nContent-Length: 0
\nSet-Cookie: NSC_wtw-XFCNBJM-FYI-443=…;expires=Tue, 16-Jun-2015 15:46:20 GMT;path=/;secure;httponly
\n
\n”)
Are there any trivial EWS requests I could try?
Your configuration may still be wrong. My service url is of the form “https://webmail/ews/exchange.asmx” (all lowercase, but I don’t think case matters).
Can you retry with:
(setq excorporate-configuration (cons “username@ourdomain.com” “https://webmail/EWS/Exchange.asmx”))
— not Services.wsdl. I think posting to the WSDL URL will give that 405 error. But again, I haven’t tested Exchange 2007 so it could still be something else going on.
Aha, it now works! Thanks so much for your generous help 🙂
Here’s the patches I’ve applied: https://gist.github.com/Wilfred/e7be1b540aa925c4e196 and here’s the 24.5 URL patch: https://gist.github.com/Wilfred/f6fc1ef185d172fdb1f0 (which is exactly the same as the 24.4 patch).
Thanks again, and let me know if you’d like me to test anything else on the Exchange 2007 server we have.
Glad to hear it’s working now.
Would you mind retrying settings autodiscovery, i.e., remove the excorporate-configuration setq, restart Emacs and retry M-x excorporate? I’m interested to know if/how it breaks on Exchange 2007.
For the next Excorporate release I’ll add Exchange 2007 and Emacs 24.5 support. If you don’t mind, I’ll add you to the Acknowledgements section, with the email address you’re posting with.
Running M-x excorporate without the setq, I’m prompted for my email address, so I supply username@ourdomain.com. I then see:
Contacting host: ourdomain.com:443
and it waits for a long time. ourdomain.com does exist, but it’s an external website and I have to configure a proxy to access outside our intranet. Perhaps the timeout is too long?
I then see (after a minute or two):
Contacting host: autodiscover.ourdomain.com:443
After another minute or two:
Excorporate: Autodiscovery ran out of URLs to try
Autodiscovery is really neat, but perhaps our environment is too customised for it to work here.
OK, thanks for checking.
Our Exchange server seems to be at or near access.london.ac.uk (I can’t easily confirm this as our IT department is Byzantine – I’m in it and I have no idea who to even ask about this) which doesn’t match any of the default patterns you test for so, I added “https://access.%s/autodiscover/autodiscover.svc” to exco–autodiscovery-templates. When I try to start Excorporate I get as stream of messages which includes this snippet:
.
.
.
Contacting host: access.london.ac.uk:443
Opening TLS connection to `access.london.ac.uk’…
Opening TLS connection with `gnutls-cli –insecure -p 443 access.london.ac.uk’…done
Opening TLS connection to `access.london.ac.uk’…done
error in process filter: let*: Symbol’s value as variable is void: url-http-response-status
error in process filter: Symbol’s value as variable is void: url-http-response-status
Contacting host: ac.uk:443
.
.
.
Excorporate: Autodiscovery ran out of URLs to try
If I try ‘gnutls-cli –insecure -p 443 access.london.ac.uk’ I do get a response which seems reasonable but I’m not sure what I should be expecting, so I don’t know. I also get something that looks okay for “openssl s_client -connect access.london.ac.uk:443 -no_ssl2 -ign_eof” but for some reason excorporate doesn’t try this line, although it does for all other autodiscover attempts.
Any thoughts?
I haven’t tested with an external gnutls-cli program. I guess you’re unable to use the superior built-in GnuTLS support, for one reason or another? You might try settings debug-on-error to ‘t to see if Emacs will give you the backtrace to see exactly where it’s failing. I couldn’t find the location from those messages.
What happens when you browse to https://access.london.ac.uk/autodiscover/autodiscover.svc in a web browser? Does it redirect you somewhere? If so, can you hack exco–autodiscovery-templates to point to the resultant URL, to avoid Emacs needing to follow redirects?
Well, I recompiled Emacs with gnutls built in (duh!) and tried various things. After tesing using https://testconnectivity.microsoft.com I set the Web Services URL to https://autodiscover-s.outlook.com/Autodiscover/Autodiscover.xml.
After M-x excorporate I was prompted for username and password but after a long pause I just got (with debugging on):
—-
Contacting host: autodiscover-s.outlook.com:443
gnutls.c: [0] (Emacs) fatal error: Error in the pull function.
—-
So I got quite a long way but fell at what felt like the last hurdle. Oh well. I suspect our screwy configuration is to blame. Thanks anyway.
Yes, autodiscovery won’t work for you, I guess, because of your server’s unfortunate configuration.
Still, if you can manage to figure out your server’s “Exchange.asmx” URL, typically something like https://mail.example.com/EWS/Exchange.asmx, you can customize the excorporate-configuration variable. Then Excorporate will skip autodiscovery. Once you find it, try browsing to https://mail.example.com/EWS/Exchange.asmx?wsdl in a web browser, and confirm you get a WSDL document after entering your username and password. If that works, Excorporate should have everything it needs.
Guys, this is great! I’ve installed after patching Hughes’ diffs and his 24.5 url patch and all works including autodiscovery. I’m on OSX Yosemite with emacs 24.5 communicating with Exchange2010_SP2. Can’t wait for more!
I tried to use it in our environment, but was not successful.
I specified excorporate-configuration to the email and service location, as autodiscovery does not work.
I believe that the problem is related to a invalid certificate of our server.
So I can access the wsdl successfully with curl only by using the ‘-k’ (and ‘–ntlm’)parameter:
curl -k –ntlm https://host/EWS/Services.wsdl -u ‘username:password’
(this returns me the wsdl)
Leaving out ‘-k’ or ”–ntlm” makes the curl call fail.
Can Excorporate handle (=ignore) invalid certificates ?
Does it use NTLM automatically ?
When you say “email and service location”, I want to first check that you used “https://host/EWS/Exchange.asmx” as the service location, not “https://host/EWS/Services.wsdl”. The latter won’t work.
If that still doesn’t work and you suspect the certificate, then try setting gnutls-log-level to 1 or 2, and check *Messages* for clues as to what’s wrong. Do you have gnutls-verify-error set to nil?
Thanks, I tried this , but it does not work neither.
I have now the impression that it is not a excorporate related problem.
I basically cannot browse any ‘https’ web page with Emacs.
So something like (browse-url-emacs “https://google.com/”) returns an empty html document.
(but no error whatsoever)
I am on Linux behind a corporate proxy, but did setup a ntlm proxy(cntlm) anmd configured emacs to use it.
So my system can can access https sides (cia chrome for example), but emacs cannot.
So I can access http websites from within emacs, but no https….
I saw in your comment in a prevoius post:
– Behind a proxy: Emacs doesn’t have CONNECT support, meaning it can’t tunnel SSL through a proxy, which breaks NTLM authentication. I think this one is fixable by adding CONNECT support to the Emacs URL package (TRAMP has it already).
Does this mean, that with Emacs there is currently for me no way to access any https website ?Using eww in emacs fails in the same way:
M-x eww https://gmail.com just returns an empty page without any error message.
Yes, HTTPS is a requirement for Excorporate, at least for all servers I’ve seen. Your M-x eww test is the right first debugging step, but it looks like you’re not past that yet.
Even when HTTPS works, you won’t be able to tunnel SSL through the proxy using Emacs (there’s currently no CONNECT support in Emacs). Because NTLM authentication needs a continuous connection, it can’t operate over a non-CONNECT tunnel.
I’ve been trying to make Excorporate pure-elisp for portability and ease of setup. I’ve never tried making it work with external tools like cntlm.
> When you say “email and service location”, I want to first check that you used “https://host/EWS/Exchange.asmx” as the service location, not “https://host/EWS/Services.wsdl”. The latter won’t work.
Our EWS URL is behind an additional auth step, so it would be really handy to have the ability to manually get the .wsdl file and pass it as a file like URL to excorporate. Have you thought about this already? What is your opinion on it?
For the next release I will improve the customization text to provide an example exchange.asmx URL. Others have pointed out that the current wording is confusing.
As for your other suggestion: did you try autodiscovery already? The server I connect to has three authentication steps and the autodiscovery algorithm correctly navigates them all. Does M-x excorporate RET with no customizations fail for you somehow? If so, then maybe the autodiscovery algorithm can be improved to handle your case, though it will be difficult for me to test that. You might have to submit a patch, which will require FSF copyright assignment paperwork.
My first paragraph was meant to be a quote from your message on the 16th of July to give context to my question. The indicator “>” has been swallowed somehow.
I will try autodiscovery and check back on progress.
I didn’t catch that it was a quote and edited the indicator out, thinking it was a garbage character. I re-added it.
When using autodiscovery, after giving the Exchange mail address, I am prompted with username and password. It then fails after a few seconds. I found in the *fsm-debug* buffer the lines:
Wed Dec 2 13:26:49 2015: exco–fsm will probe https://autodiscover.host.domain/autodiscover/autodiscover.svc
Wed Dec 2 13:26:59 2015: exco–fsm didn’t find https://autodiscover.host.domain/autodiscover/autodiscover.svc
but I am able to connect to https://autodiscover.host.domain/autodiscover/autodiscover.svc via the browser and after Auth there (using the same user/pass as when prompted after M-x excorporate), I get the https://autodiscover.host.domain/autodiscover/Services.wsdl alright. How can I debug this further?
Interesting, I haven’t seen that case yet. Try:
emacs -Q
M-: (require ‘excorporate) RET
M-: (setq url-debug ‘t) RET
M-: (url-retrieve-synchronously “https://autodiscover.host.domain/autodiscover/autodiscover.svc”) RET
And check the *URL-DEBUG* buffer for hints as to what might be going wrong.
The *URL-DEBUG* buffer looks good to me: http://pastebin.com/w2nP1NCk
Yes, everything looks OK, except for the final 401. I compared this output to a working request for autodiscover.svc in my setup, and the only minor differences are:
– I have cookies enabled
– I get:
Got 0-length content-length, activating callback immediately.
whereas you get:
Calling initial content-length for extra data at end of headers
– I get ^M line endings, whereas you get double-newlines, but I suspect that’s just pastebin munging of some sort
It seems the server is returning non-zero content for you. I guess you can double-check that you entered the correct credentials (but I assume you tried a few times), and maybe try another version of Emacs. You could also try enabling cookies in Emacs but I don’t think that should affect authentication. The other thing to do is compare in detail the browser-back-n-forth with what Emacs is doing.
Ok, I enabled cookies, just to be on the safe side.
I looked into it in more detail. When I run
(url-retrieve-synchronously “https://autodiscover.host.domain/autodiscover/autodiscover.svc”)
then when Emacs prompts for authentication, the *URL-DEBUG* buffer reads as http://pastebin.com/ddJvGWCT.
The way I see it, there is some auth-related stuff going on even before the expected prompt for an username. Could this be related to the “Calling initial content-length for extra data at end of headers” difference you mentioned?
Also, on a possibly related note, when visiting https://autodiscover.host.domain/autodiscover/autodiscover.svc in the browser and getting the popup prompt for authentication, pressing ESC re-prompts me for auth, with a resized auth popup window. Could this indicated different auth-types, i.e. with fallbacks as in “http -> Handling normal authentication” before actual auth prompt in Emacs?
Finally, I tried to make sure there is no other auth process going on in the background / in parallel, e.g. .authinfo.gpg or similar.
To sum up, I am at a loss as to how to investigate further…
The *URL-DEBUG* output before you start authenticating seems to be as-expected, because Emacs sees the 401, then prompts you for credentials.
Can you try with this patch to url-http.el:
— a/lisp/url/url-http.el
+++ b/lisp/url/url-http.el
@@ -886,6 +886,7 @@ url-http-parse-headers
(url-handle-content-transfer-encoding))
(url-http-debug “Finished parsing HTTP headers: %S” success)
(widen)
+ (url-http-debug “Response: %s” (buffer-string))
(goto-char (point-min))
success))
After you restart Emacs to start a clean test, make sure to re-evaluate url-http-parse-headers with this change, otherwise it won’t take because url-http-parse-headers is compiled and built-in.
Then check the full responses you get in *URL-DEBUG*. Maybe you can do the same thing in the browser with Firebug or another tool, and compare the responses it gives. The latter will give you more information about what is happening with the “ESC fallback auth method” as well. Maybe Basic (non-NTLM) authentication will work on this server, in which case deregistering the NTLM URL scheme in Emacs might work.
Thanks for the ideas.
The session log for Firefox before auth is http://pastebin.com/g673eKv0
The session log for Firefox after auth is http://pastebin.com/DXuuq02p
The session log for Emacs with new debug info is http://pastebin.com/amFgtZcy
OK, I looked at those NTLM strings. You didn’t provide the Type 3 message in the Firefox case (the last request, with the third (long) NTLM string). One thing you could try in Emacs is customizing ntlm-compatibility-level, setting it 0. Then retry M-x excorporate.
The Firefox Type 3 message might help me figure out what your server is expecting.
I was using the “Console” log tab when I should have used the more obvious “Network” log tab… so the Type 3 message is http://pastebin.com/4W70chmj
Also, (setq ntlm-compatibility-level 0) did not change anything.
It looks like the Firefox Type 3 target information block has extra information in it, compared to what comes back in the Type 2 message, including “HTTP/autodiscover.host.domain”. Emacs doesn’t send that extra information, and I suspect that’s why the server rejects its Type 3 message. The problem is I have no idea how to generate that extra information, and I’ve checked my usual NTLM source references.
The only thing I can suggest is talking to your Exchange administrator and seeing if there’s a staging server that you can experiment with server-side settings on, in an attempt to make the server more permissive.
On Windows, Firefox supports NTLMv2 by using the Windows proprietary built-in NTLM implementation. I suspect that you would not be able to retrieve the autodiscover.svc URL on Firefox on a GNU/Linux machine. That might be worth a try though, just to see if it works.
I will contact the Exchange administrator for a more detailed description of server-side settings.
Also, I followed your suggestion and tried to retrieve the autodiscover.svc URL with a Firefox nightly build on a very recent GNU/Linux system and it succeeds – another browser on that system, in this case Konqueror, did not succeed, though.
Oh, if Firefox on GNU/Linux works, then there are more options before contacting the administrator. Maybe I’ll be able to fix Emacs’s NTLM library to work in your case. Can you send the network log of Firefox on GNU/Linux?
The network log of Firefox on GNU/Linux shows 4 GET methods, with status 401, 401, 302, 200 respectively and detailed headers:
GET 401: http://pastebin.com/P5wg1sjK
GET 401: http://pastebin.com/7Q9cx3He
GET 302: http://pastebin.com/uPMdkmg4
GET 200: http://pastebin.com/q3JeGcWy
There are a few differences between what Firefox and Emacs are sending in the Type 3 response. We might need to try a few of them. We can start with the most likely cause, which is that Emacs is sending a blank LMv2 field.
After (require ‘excorporate), can you make this change to ntlm-build-auth-response and re-evaluate it, then retry M-x excorporate?
@@ -231,14 +231,14 @@ ntlm-build-auth-response
targetInfo ; target info
(make-string 4 0))) ; unknown
;; for reference: LMv2 interim calculation
– ;; (lm-interim (hmac-md5 (concat challengeData nonce)
– ;; ntlmv2-hash))
+ (lm-interim (hmac-md5 (concat challengeData nonce)
+ ntlmv2-hash))
(nt-interim (hmac-md5 (concat challengeData blob)
ntlmv2-hash)))
;; for reference: LMv2 field, but match other clients that
;; send all zeros
– ;; (setq lmRespData (concat lm-interim nonce))
– (setq lmRespData (make-string 24 0))
+ (setq lmRespData (concat lm-interim nonce))
+ ;; (setq lmRespData (make-string 24 0))
(setq ntRespData (concat nt-interim blob))))
;; compatibility level is 2, 1 or 0
;; level 2 should be treated specially but it’s not clear how,
I applied the patch and rerun, *URL-DEBUG* output is http://pastebin.com/2VeDFvkC, from user perspective behaviour is unchanged.
OK. I did some more testing, and I’m wondering if this is related to the Type 3 workstation field. There are other Type 3 differences too though. I have a patch that should make the Emacs Type 1 and Type 3 requests identical to the one Firefox is sending for you — but the patch doesn’t work against my server. I’ll upload it somewhere later today for you to try. But in the meantime, can you try on GNU/Linux configuring the Firefox preference “network.generic-ntlm-auth.workstation” to something other than WORKSTATION and see if authentication fails?
Can you email me at _this blog domain base name_ at _this blog domain base name_.org ? Then we can exchange patch files and logs more easily.
Just to follow up here, after lots of back and forth, we resolved Fabio’s authentication issue. However, the fix makes significant changes to ntlm.el; I’m not sure how those changes would affect other Excorporate users. I’ll have to think about how to make those changes more widely available.
Hi Thomas
Any plans to release a new version, at least on here if not on GNU ELPA? This is a fantastic project and I’d love to see it generally available.
Hi Wilfred,
Yes, I plan to release newer versions through GNU ELPA. I did release 0.6.1 recently here. Its only change was the addition of Emacs 24.5 support:
https://www.fitzsim.org/emacs/excorporate-0.6.1.tar
I didn’t do a full blog post for it, just mentioned it in one of these comment threads. I guess you also need Exchange 2007 support to smooth out your installation. Maybe I’ll do a 0.6.2 here for that as well.
Right now I’m working on fixing GNU ELPA’s build scripts to support including core packages from Emacs in a maintainable way. And I got soap-client.el and ntlm.el into good shape before Emacs 25.1 branched, so that will be two fewer things that Excorporate users will need to download.
One bump I ran into is that calfw has a bunch of contributors who haven’t signed FSF paperwork. I’ll try to work with calfw’s maintainer to get all the paperwork done but in the meantime I might have to remove calfw support and write an Org-mode Agenda front-end. Org-mode is already in Emacs so it won’t block Excorporate getting into GNU ELPA.
Once everything is properly upstreamed I’ll get back to adding a few more features. The Org-mode support (which I planned to do anyway) will help make sure the API is in good shape.
Thomas