[Buildroot] Unicode problem with check-uniq-files
Yann E. MORIN
yann.morin.1998 at free.fr
Thu Mar 22 20:41:45 UTC 2018
Jaap, All,
Please, keep the list in Cc next time...
On 2018-03-22 11:56 +0100, Jaap Crezee spake thusly:
> On 03/22/18 11:43, Jaap Crezee wrote:
> > ./support/scripts/check-uniq-files -t target /data/work/jcz/git/jidiot/clients/innr/buildroot_development/output/build/packages-file-list.txt
> > Traceback (most recent call last):
> > File "./support/scripts/check-uniq-files", line 42, in <module>
> > sys.exit(main())
> > File "./support/scripts/check-uniq-files", line 31, in main
> > for row in r:
> > File "/usr/lib/python3.6/encodings/ascii.py", line 26, in decode
> > return codecs.ascii_decode(input, self.errors)[0]
> > UnicodeDecodeError: 'ascii' codec can't decode byte
>
> Attached patch is working for me. If you agree with it, you can apply it.
> If you like I can ack. If you do not agree with this patch, what do you suggest?
> diff --git a/support/scripts/check-uniq-files b/support/scripts/check-uniq-files
> index be808cce03..82b0af24ba 100755
> --- a/support/scripts/check-uniq-files
> +++ b/support/scripts/check-uniq-files
> @@ -26,7 +26,7 @@ def main():
> return False
>
> file_to_pkg = defaultdict(list)
> - with open(args.packages_file_list[0], 'r') as pkg_file_list:
> + with open(args.packages_file_list[0], 'r', encoding="utf-8") as pkg_file_list:
> r = csv.reader(pkg_file_list, delimiter=',')
> for row in r:
> pkg = row[0]
I'll be testing that, but it has to work in quite a few situations:
- python 2.6, python 2.7, python 3.x
- current locale is UTF-8 (is it LANG, or any of the other LC_* ones?)
or it is not an UTF-8 locale.
However, we already discussed this with Thomas on IRC the other day, and
nothing guarantees that filenames are stored as UTF-8 streams on disk.
Since packages-file-list.txt only contains whatever 'find' will put in
there, and that 'find' will only put whatever it sees on-disk, its
encoding is definitely unpredictable, probably depending on the user's
configuration.
So, even if UTF-8 is the prevalent encoding, nothing guarantees that it
is the only one we'd ever see, AFAIU...
Which means that your solution is probably just only a workaround that
happens to work for you and a lot of other situations, but is not the
correct solution.
I've been hacking that check-uniq-file script for two evenings now, and
I still don't see a good solution that makes it work in both python2 and
python3, with an UTF-8 locale or not...
I was thinking that maybe we could make it a python2 (not python)
script, but then some distros are switching to a python3-only setup
now, so that would break on those distros... Do you use such a distro,
by chance? Which one?
Anyway, more testing to be done here, thanks for the suggestion. I'll
report back later...
Regards,
Yann E. MORIN.
--
.-----------------.--------------------.------------------.--------------------.
| Yann E. MORIN | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software Designer | \ / CAMPAIGN | ___ |
| +33 223 225 172 `------------.-------: X AGAINST | \e/ There is no |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL | v conspiracy. |
'------------------------------^-------^------------------^--------------------'
More information about the buildroot
mailing list