<div dir="ltr"><div class="gmail_default" style="font-family:tahoma,sans-serif"><br></div><div class="gmail_extra"><br><div class="gmail_quote">On 1 December 2017 at 16:19, Florian Bruhin <span dir="ltr"><<a href="mailto:me@the-compiler.org" target="_blank">me@the-compiler.org</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">(Re-adding the mailinglist again, until someone there tells me they've heard<br>
enough horror stories about encodings :D)<br>
<span class="gmail-"><br>
On Fri, Dec 01, 2017 at 01:46:39PM +0000, J Barchan wrote:<br>
> On 1 December 2017 at 12:18, Florian Bruhin <<a href="mailto:me@the-compiler.org">me@the-compiler.org</a>> wrote:<br>
><br>
> ><br>
> ><br>
> > Yes, thinking about it again, I was wrong there - sorry! If you can't force<br>
> > robocopy to output UTF-8, the best (and probably correct) guess is<br>
> > <br>
> > locale.getpreferredencoding():<br>
> > <a href="https://docs.python.org/3/library/locale.html#locale.getpreferredencoding" rel="noreferrer" target="_blank">https://docs.python.org/3/<wbr>library/locale.html#locale.<wbr>getpreferredencoding</a><br>
> ><br>
> > The documentation for subprocess mentions that it's using<br>
> > locale.getpreferredencoding(<wbr>False) if no encoding was given. I'm still<br>
> > not sure<br>
> > what that argument (do_setlocale) means exactly, but you might want to<br>
> > just do<br>
> > the same.<br>
> ><br>
><br>
> Dear Florian,<br>
><br>
</span>> This is *almost* what I was always looking for, which nobody has come up<br>
<span class="gmail-">> with till now...!!! But read on....<br>
><br>
> All the people elsewhere who have been saying use utf-8 or utf-16 or the<br>
> file system encoding or all the other things have been doing my head in...<br>
> :(<br>
<br>
</span>Can't blame them for the wrong answers, this stuff is hard.<br>
If you thought it this was bad enough, things going to get worse in the rest of<br>
this mail.<br>
<br>
You were asking how other tools manage to do this (like Visual Studio).<br>
I haven't tried, but either they've been written by people who fully understand<br>
this madness, or they do the wrong thing. Probably the latter. :D<br>
<span class="gmail-"><br>
> After I had penned my long epistle to you earlier, I discovered that,<br>
</span>> although some of the solutions had *appeared* to work in that I no longer<br>
> got the conversion error, it turns out they were then *displaying* that £<br>
<span class="gmail-">> character differently, e.g. as the "oe" character or other symbol.<br>
<br>
</span>See? There's no easy answer if you want a correct result ;-)<br>
<br>
> I discovered for myself that *only* decode("cp850") caused it to display as<br>
> the desired £. And that's because UK/Western Europe is *Code Page 850*.<br>
<span class="gmail-">> So it all fits together now!<br>
><br>
</span>> Now, I *assumed* your suggestion of locale.getpreferredencoding() would<br>
<span class="gmail-">> return cp850 under Windows (remember, I do not have Windows python to<br>
> test). But I checked with my Windows stakeholder, and it does not --- it<br>
</span>> returns the "windows_1252" type string, which we know does *not* get the £<br>
<span class="gmail-">> right....<br>
><br>
> Then, I got him to open a Command Prompt, and just enter the command:<br>
><br>
</span>> c:\Tmp> *chcp*<br>
<span class="gmail-">> > Active code page: 850<br>
> ><br>
><br>
> That response from chcp of 850 as the CP is precisely what I am looking<br>
> for....!!!<br>
><br>
</span>> *Soooo*, I have a little request-task for you! Assuming you have Windows +<br>
<span class="gmail-">> Python 3, can you find a native Qt/PyQt/Python call which returns that<br>
> under Windows, please, please?? If you're not UK, obviously your CP value<br>
> may vary, but you know what I'm seeking. Thank you so much!<br>
<br>
</span>So... further down the rabbit hole. I've learned things about Windows and<br>
encodings, and I wish I didn't look :D<br>
<br>
Codepage 850 is apparently what Windows (or rather, DOS) used before there was<br>
windows-1251 (which in theory is *also* replaced by an Unicode encoding<br>
nowadays, but yeah... in theory).<br>
<br>
Money quote from <a href="https://en.wikipedia.org/wiki/Code_page_850" rel="noreferrer" target="_blank">https://en.wikipedia.org/wiki/<wbr>Code_page_850</a> :<br>
<br>
> Systems largely replaced code page 850 with, first, Windows-1252 (often<br>
> mislabeled as ISO-8859-1), and later with UCS-2, and finally with UTF-16.<br>
<br>
Note it was introduced in 1987...<br>
<br>
Anyways - locale.getpreferredencoding() returns windows-1252 because that's<br>
what the part of Windows which is a bit less ancient uses. Turns out the<br>
console is even *more* horrible than that.<br>
<br>
Can you please try this (Windows only):<br>
<br>
import ctypes<br>
ctypes.cdll.kernel32.<wbr>GetConsoleOutputCP()<br>
<br>
See <a href="https://docs.microsoft.com/en-us/windows/console/getconsoleoutputcp" rel="noreferrer" target="_blank">https://docs.microsoft.com/en-<wbr>us/windows/console/<wbr>getconsoleoutputcp</a><br>
<br>
BUT! We're not done there yet, things get *more* horrible. (Yes, I didn't know<br>
that was possible either).<br>
<br>
That returns "437" for me. That's is an *even older* codepage, from 1981.<br>
It *does* happen to have the pound sign at the right position though :D<br>
<br>
See <a href="https://en.wikipedia.org/wiki/Code_page_437" rel="noreferrer" target="_blank">https://en.wikipedia.org/wiki/<wbr>Code_page_437</a><br>
<br>
I originally wanted to blame Robocopy for doing horrible things, but it's<br>
really Windows in general:<br>
<br>
>>> p = subprocess.run(['cmd', '/C', 'echo £'], stdout=subprocess.PIPE)<br>
>>> p.stdout<br>
b'\x9c\r\n'<br>
>>> p.stdout.decode('cp850')<br>
'£\r\n'<br>
>>> p.stdout.decode('cp437')<br>
'£\r\n'<br>
<br>
What happens with characters which aren't in cp437 is left as an exercise to<br>
the reader, I have no idea.<br>
<span class="gmail-"><br>
> BTW, in <a href="http://doc.qt.io/qt-5/qprocess.html" rel="noreferrer" target="_blank">http://doc.qt.io/qt-5/<wbr>qprocess.html</a> I do not see any mention of<br>
> "encoding at all"? Where were you getting your second paragraph above from?<br>
<br>
</span>I was looking at Python's subprocess module:<br>
<a href="https://docs.python.org/3/library/subprocess.html" rel="noreferrer" target="_blank">https://docs.python.org/3/<wbr>library/subprocess.html</a><br>
<br>
QProcess is probably the better choice if you're running things from a GUI,<br>
though.<br>
<br>
So yeah, I'd recommend something like:<br>
<br>
if sys.platform == 'win32':<br>
encoding = ctypes.cdll.kernel32.<wbr>GetConsoleOutputCP()<br>
else:<br>
encoding = locale.getpreferredencoding(<wbr>False)<br>
<br>
And now I'll need to do something other than reading about encodings for a<br>
while... :D<br>
<div class="gmail-HOEnZb"><div class="gmail-h5"><br>
Florian<br>
<br>
--<br>
<a href="https://www.qutebrowser.org" rel="noreferrer" target="_blank">https://www.qutebrowser.org</a> | <a href="mailto:me@the-compiler.org">me@the-compiler.org</a> (Mail/XMPP)<br>
GPG: 916E B0C8 FD55 A072 | <a href="https://the-compiler.org/pubkey.asc" rel="noreferrer" target="_blank">https://the-compiler.org/<wbr>pubkey.asc</a><br>
I love long mails! | <a href="https://email.is-not-s.ms/" rel="noreferrer" target="_blank">https://email.is-not-s.ms/</a><br>
</div></div></blockquote></div><br><br clear="all"><div style="font-family:tahoma,sans-serif" class="gmail_default">Hi Florian,</div><div style="font-family:tahoma,sans-serif" class="gmail_default"><br></div><div style="font-family:tahoma,sans-serif" class="gmail_default">You're a hero! On mine, <span style="font-family:monospace,monospace">ctypes.cdll.kernel32.GetConsoleOutputCP()</span> does indeed return <span style="font-family:monospace,monospace">850</span> !! (I only just reluctantly installed Python on my Windows a few minutes ago, else I couldn't have done this!) So it's considered something not worthy of being available natively from Python, without going down the <span style="font-family:monospace,monospace">cdll.kernel32</span> rabbit-hole, I wonder why? ;-)</div><div style="font-family:tahoma,sans-serif" class="gmail_default"><br></div><div style="font-family:tahoma,sans-serif" class="gmail_default">I've been thinking about all this too, in view of how messy it is.</div><div style="font-family:tahoma,sans-serif" class="gmail_default"><br></div><div style="font-family:tahoma,sans-serif" class="gmail_default">I'm coming to the likely conclusion that this issue & its solution are all to do with when running "DOS" (deliberately in quotes) commands, only. We know <span style="font-family:monospace,monospace">dir</span> or <span style="font-family:monospace,monospace">echo</span> are only internal-aliases within <span style="font-family:monospace,monospace">cmd.exe</span>, and while <span style="font-family:monospace,monospace">robocopy</span> is a genuine external program it's probably a DOS-sy rather than a Windows-y program. (I don't know what that means exactly, but it'll probably be to do with how the executable is marked "Windows" or not, I recall that being some kind of option when from the linker.)</div><div style="font-family:tahoma,sans-serif" class="gmail_default"><br></div><div style="font-family:tahoma,sans-serif" class="gmail_default">Now I think all this dealing with <span style="font-family:monospace,monospace">£</span> character as 0x9c --- which is the root of the problem, and should really be a "oe" character anyway --- is at the DOS side, because for example if I insert a <span style="font-family:monospace,monospace">£</span> into Notepad and save the file it is stored as a 0xa9 if ANSI or as 0xc2,0xa3 if UTF-8. So I'm thinking that the whole <span style="font-family:monospace,monospace">cp850</span> Code Page has nothing to do with Windows-Unicode-or-whatever-level, and is only coming into play because I'm using things which happen to still be DOS-sy, such as <span style="font-family:monospace,monospace">cmd.exe</span> or <span style="font-family:monospace,monospace">robocopy.exe</span>.</div><div style="font-family:tahoma,sans-serif" class="gmail_default"><br></div><div style="font-family:tahoma,sans-serif" class="gmail_default">I guess the filename is not <i>really</i> being stored with an 0x9c in it, but some Windows-Unicode-or-other-character, but it appears to "DOS-sy" programs like it is a 0x9c. If they were true, native "Windows" applications they wouldn't be seeing or outputting 0x9c for the <span style="font-family:monospace,monospace">£</span>, but some Unicode or whatever, and none of this would have arisen.</div><div style="font-family:tahoma,sans-serif" class="gmail_default"><br></div><div style="font-family:tahoma,sans-serif" class="gmail_default"><i>However</i>, "true Windows-y programs" which can be invoked purely from the command-line, don't show a GUI, and report the files they are backing up on stdout/stderr as they progress for reading into my window <i>probably</i> are not going to be very common... :)</div><div style="font-family:tahoma,sans-serif" class="gmail_default"><br></div><div style="font-family:tahoma,sans-serif" class="gmail_default">[ BTW, I do have to use Qt <span style="font-family:monospace,monospace">QProcess</span> for spawning, I use the <span style="font-family:monospace,monospace">readyReadStandardOutput </span>signal to be fetching the output progress filenames into the scrollable window for my user to see as it goes along, the Python <span style="font-family:monospace,monospace">os</span> module is not going to let me do that. This is why I'm posting to a PyQt forum! ]<br></div><div style="font-family:tahoma,sans-serif" class="gmail_default"><br></div><div style="font-family:tahoma,sans-serif" class="gmail_default">This whole thing has done my head in. If you have further observations, or agree or disagree with what I am saying above, let me know, I'm always interested.</div><div style="font-family:tahoma,sans-serif" class="gmail_default"><br></div><div style="font-family:tahoma,sans-serif" class="gmail_default">Thank you <i>so much</i> for your time, effort & explanations. I respect people who know their stuff. Doubtless you'll hear from me with other PyQt topics in due course, you have been warned... :) Have a good weekend.</div><div style="font-family:tahoma,sans-serif" class="gmail_default"><br></div><div style="font-family:tahoma,sans-serif" class="gmail_default">Kindest,</div><div style="font-family:tahoma,sans-serif" class="gmail_default">Jonathan<br></div><div style="font-family:tahoma,sans-serif" class="gmail_default"><br></div><div style="font-family:tahoma,sans-serif" class="gmail_default"><br></div></div></div>