Pre Delphi 2008-9 Unicode Do's and Dont's

[Lee Jenkins]

Has anyone posted information concerning do's and dont's for Unicode support in
upcoming Delphi versions?

It recent threads concerning Delphi/Unicode, I think the topic of being prepared
for Unicode has not been addressed so much, at least as far as I can see.

On one side, we have applications that have already been written whose authors
are rightfully concerned about compatibility.

On the other side, we have applications which are yet to be written and do not
have much threat of being

In the middle, we have applications which are currently being written (raises
hand) which could benefit from some suggestions on best practices to give the
applications currently being written to have a chance of being ported more
easily when D2008/9 is finally released.

--
Warm Regards,

Lee

[Nick Hodges (Embarcadero)]

Post by Lee Jenkins
Has anyone posted information concerning do's and dont's for Unicode
support in upcoming Delphi versions?

I'll be posting some articles on this very soon.

Short list:

Don't assume that the size of a Char is one.

Don't assume that the size of an array of Char is the same as the
Length of the string held in the array of Char.

[John Herbster]

Post by Nick Hodges (Embarcadero)

Post by Lee Jenkins
Has anyone posted information concerning do's and dont's for
Unicode support in upcoming Delphi versions?

I'll be posting some articles on this very soon.

Nick, Here are a few suggestions and clarifications.

Post by Nick Hodges (Embarcadero)
Don't assume that the size of a Char is one.

Please start with the compiler op to switch def of "Char".

Post by Nick Hodges (Embarcadero)
Don't assume that the SizeOf an array of Char is the same as the
Length of the string held in the array of Char [less one].

Show us how to iterate through a string of characters with indexes.

Show us how to iterate through a string of characters with pointers.

Show us how to load and store a string from and to TStreams.

Show us how to replace a character.

Show us how to make literal constants an assign them to strings.

Show us how to pass strings to and from DLLs.

Regards, JohnH

[Nick Hodges (Embarcadero)]

Post by John Herbster
Show us how to iterate through a string of characters with indexes.

Exactly as before.

Post by John Herbster
Show us how to iterate through a string of characters with pointers.

Exactly as before -- but don't assume a character is of size 1.

Post by John Herbster
Show us how to load and store a string from and to TStreams.

Exactly as before but you can't assume that the length of a string char
is 1.

Post by John Herbster
Show us how to replace a character.

Exactly as before.

Post by John Herbster
Show us how to make literal constants an assign them to strings.

Exactly as before.

Post by John Herbster
Show us how to pass strings to and from DLLs.

Just as before, but again, don't assume that Char = 1 byte.

[John Herbster]

Post by Nick Hodges (Embarcadero)
Exactly as before -- but don't assume a character is of size 1.

Thanks Nick!

Post by Nick Hodges (Embarcadero)

Post by John Herbster
Show us how to iterate through a string of characters with pointers.

Exactly as before -- but don't assume a character is of size 1.

May I presume like this?
p := @MyString[1];
Inc(p);
where MyStr: string; and p: PChar;

And how expensive are these operations during CPU execution?

TIA, JohnH

[Nick Hodges (Embarcadero)]

Post by John Herbster
May I presume like this?
Inc(p);
where MyStr: string; and p: PChar;

Yes -- just like before.

Post by John Herbster
And how expensive are these operations during CPU execution?

Minimal -- it's very efficient. It's pointer math, right? ;-)

[Paul Scott]

On Mon, 21 Jul 2008 19:45:44 +0100, Nick Hodges (Embarcadero)

Post by Nick Hodges (Embarcadero)

Post by John Herbster
May I presume like this?
Inc(p);
where MyStr: string; and p: PChar;

Yes -- just like before.

Err... Nick,

Doesn't this depend on exactly what you mean by "a character" ?


If CG had gone for UTF-32 encoding, there would have indeed have been a
one-to-one correspondence between a position in the string and what most
people would think of as "a character" - ie. a CodePoint.

But since (according to the blogs) Tiburon will be using "string=UTF-16",
then any character (CodePoint) which lies outside the BMP (ie. >64K) has
to be represented by a /pair/ of 16-bit CodeValues.

So, since your blog also says that for Tiburon "Char=WideChar=word", one
external "character" may actually need two internal "characters" to
represent it.

Now while most Delphi programs hopefully won't ever see any of these
million or so extended CodePoints, anywhere that a program allocates
memory on the basis of "ExpectedNumberOfChars x SizeOf(Char)" or, like the
code snippet above, might presume that each index position in a "string"
represents "one character" in some writing system leaves a ticking
timebomb.

[Thorsten Engler [NexusDB]]

Post by Paul Scott
Doesn't this depend on exactly what you mean by "a character" ?

Exactly the same thing that an AnsiString meant by "a character". Where
you could have DBCS and MBCS.

Post by Paul Scott
But since (according to the blogs) Tiburon will be using
"string=UTF-16", then any character (CodePoint) which lies outside
the BMP (ie. >64K) has to be represented by a pair of 16-bit
CodeValues.

UTF-16 is the native type for unicode character data on the Windows
platform. Any Windows API taking or returning unicode strings uses
UTF-16. Internally ALL string data under NT based Windows versions (NT,
2000, and newer) is using UTF-16. All the ANSI APIs still offered are
just wrappers which perform the conversion into UTF-16 and call the
actual native API.

The use of any other encoding as default for unicode strings on Windows
does not make sense.

Post by Paul Scott
So, since your blog also says that for Tiburon "Char=WideChar=word",
one external "character" may actually need two internal "characters"
to represent it.

Again, this is no different at all from DBCS and MBCS ANSI encodings.

The huge difference is that in UTF-16 the value ranges for singelton,
leading and trailing units do NOT overlap. That means that in almost
all cases you can ignore the fact that UTF-16 contains surrogate pairs.

In DBCS/MBCS ANSI you have to parse the strings always starting from
the beginning because you might need to skip ahead if you hit a lead
byte because one of the trailing bytes might have a value that on it's
own would represent a valid singelton (with a totally different meaning
then this trailing byte in combination with it's lead byte).

In UTF-16 a trailing unit seen on it's own can never ever be mistaken
for a different singelton value.

Post by Paul Scott
Now while most Delphi programs hopefully won't ever see any of these
million or so extended CodePoints, anywhere that a program allocates
memory on the basis of "ExpectedNumberOfChars x SizeOf(Char)" or,
like the code snippet above, might presume that each index position
in a "string" represents "one character" in some writing system
leaves a ticking timebomb.

Again, Delphi doesn't have any more or less problems with this then any
other application on the Windows platform. UTF16 is the native unicode
encoding on windows. Using UTF-16 is much much less of a problem then
trying to support ANSI for DBCS or MBCS codepages.

--

[Paul Scott]

On Tue, 22 Jul 2008 13:04:48 +0100, Thorsten Engler [NexusDB]

Post by Thorsten Engler [NexusDB]
Again, Delphi doesn't have any more or less problems with this then any
other application on the Windows platform. UTF16 is the native unicode
encoding on windows. Using UTF-16 is much much less of a problem then
trying to support ANSI for DBCS or MBCS codepages.

Thorsten,

I don't disagree with anything you said - and it was probably all obvious
to those who have been struggling with multiple code pages in the past.

But for the rest of us with applications in "English", there's more to
"Unicodifying an Application" than just a recompilation - even with a
liberal sprinkling of "SizeOf(Char)"

[Allen Bauer (CodeGear)]

Post by Paul Scott
On Tue, 22 Jul 2008 13:04:48 +0100, Thorsten Engler [NexusDB]

Post by Thorsten Engler [NexusDB]
Again, Delphi doesn't have any more or less problems with this then
any other application on the Windows platform. UTF16 is the native
unicode encoding on windows. Using UTF-16 is much much less of a
problem then trying to support ANSI for DBCS or MBCS codepages.

Thorsten,
I don't disagree with anything you said - and it was probably all
obvious to those who have been struggling with multiple code pages
in the past.
But for the rest of us with applications in "English", there's more
to "Unicodifying an Application" than just a recompilation - even
with a liberal sprinkling of "SizeOf(Char)"

If all your application ever expected to get was "English," just
because strings now support a much larger set of codepoints doesn't
suddenly mean that your application will magically begin to get all
kinds of characters outside the normal English characters in the ASCII
range. Only if your application were deployed to some region where
there is the possibility of non-English characters being encountered
will you run in to an issue. Rebuilding an application should not alter
its existing use-cases.

So as long as your application continues to get deployed into
environments that never encounter non-English characters, things should
just continue to work.

[Q Correll]

Allen,

| So as long as your application continues to get deployed into
| environments that never encounter non-English characters, things should
| just continue to work.

Where can I get a pair of those rose-colored glasses? ;-)

[Allen Bauer (CodeGear)]

Post by Q Correll
Allen,

Post by Allen Bauer (CodeGear)
So as long as your application continues to get deployed into
environments that never encounter non-English characters, things
should just continue to work.

Where can I get a pair of those rose-colored glasses? ;-)

My point was that just because you have a potentially more capable
application doesn't mean that it will magically begin to use that
capacity when faced with same use-cases and scenarios it always has. If
a user always just entered English string data into an application,
having a new version of that application that accepts a wider range of
characters will not mean that this same user will just being to enter
(or even know how to enter) non-English data.

[Q Correll]

Allen,

Yes, I understand.

But my "point" was that things rarely seem to work out as simply as we
expect. <g>

[Thorsten Engler [NexusDB]]

Post by Q Correll
Allen,
Yes, I understand.
But my "point" was that things rarely seem to work out as simply as
we expect. <g>

I have to say I'm with Allen here, if your application has only been
confronted with standard english characters in the past then just
switching out that application in the same environment with one which
is unicode enabled you are still only going to be confronted with
standard english characters.



--

[Q Correll]

Thorsten,

| I have to say I'm with Allen here, if your application has only been
| confronted with standard english characters in the past then just
| switching out that application in the same environment with one which
| is unicode enabled you are still only going to be confronted with
| standard english characters.

I yield to the gurus! <g>

[Ray Porter]

Post by Thorsten Engler [NexusDB]
I have to say I'm with Allen here, if your application has only been
confronted with standard english characters in the past then just
switching out that application in the same environment with one which
is unicode enabled you are still only going to be confronted with
standard english characters.

One thing I haven't heard made clear yet is possible impact on database
reads/writes. We use Oracle 10G and will eventually move to SQL Server.
I'll ask our DBA but I'm fairly certain our database is currently configured
for ansi (Latin char set). Will I need to ansify our database reads/writes
(or at least the writes) or will the TADODataset and its descendents handle
the situation transparently?

Thanks,
Ray Porter

[Thorsten Engler [NexusDB]]

Post by Ray Porter
One thing I haven't heard made clear yet is possible impact on database
reads/writes. We use Oracle 10G and will eventually move to SQL Server. I'll
ask our DBA but I'm fairly certain our database is currently configured for
ansi (Latin char set). Will I need to ansify our database reads/writes (or
at least the writes) or will the TADODataset and its descendents handle the
situation transparently?

All that should just continue to work. You just keep using TField.AsString to
read/write and the classes will take care that whatever ends up in the database
does so in the right format.


--

[willr]

Post by Q Correll
Allen,
| So as long as your application continues to get deployed into
| environments that never encounter non-English characters, things should
| just continue to work.
Where can I get a pair of those rose-colored glasses? ;-)

I think they are available at most programming classes -- and "extra
rose coloured" are available at "design" classes.

[Q Correll]

willr,

| I think they are available at most programming classes -- and "extra
rose coloured" are available at "design" classes.

<chuckle>

Regardless of what Allen and Thorsten feel will be the case, I'd still put
money on the phenomenon of "unintended consequences" popping up. <g> It
may not be with the display of any given character set, but my gut says
something unexpected will surface. Eight million lines of code is very
difficult to keep in the box when one starts messing with it. ;-)

[Thorsten Engler [NexusDB]]

Post by Q Correll
Regardless of what Allen and Thorsten feel will be the case, I'd
still put money on the phenomenon of "unintended consequences"
popping up. <g> It may not be with the display of any given
character set, but my gut says something unexpected will surface.
Eight million lines of code is very difficult to keep in the box when
one starts messing with it. ;-)

I never said that porting to code from ANSI to Unicode will be without
any issues and gotchas.

But that is NOT what Allen was talking about.

What he and I was saying is that if your current ANSI application is
only confronted with simple, single byte characters (coming from Win
API calls, or typed in by the user and so on), then converting your
application to Unicode will NOT suddenly expose you to things like
surrogate pairs.

Only if the environment in which your application runs is changed could
it find itself be confronted with surrogate pairs.

But in any single possible case where your unicode application will be
confronted with surrogate pairs, the same application as ANSI
application would be confronted with double or even multi byte
character sets. Which are much more difficult to handle then Unicode
with surrogate pairs.


--

[Q Correll]

Thorsten,

| But that is NOT what Allen was talking about.

I know, I know. <g> Allen used the noun "things." I was just having a
bit of "fun" jabbing at his high optimism. I fully realize the "things"
Allen meant were related to the display of the characters. But there may
be some, if not many, other things that may be affected by the changes.
Especially until Tiburon is significantly debugged. ;-)

[Thorsten Engler [NexusDB]]

Post by Q Correll
I know, I know. <g> Allen used the noun "things." I was just
having a bit of "fun" jabbing at his high optimism. I fully realize
the "things" Allen meant were related to the display of the
characters. But there may be some, if not many, other things that
may be affected by the changes. Especially until Tiburon is
significantly debugged. ;-)

Read Allen's post again. He was specifically talking about what types
of characters your code might get confronted with.

Supporting Unicode is *not* the *cause* for being confronted with new
types of characters.

If, because of some external change, your application should be
confronted with characters it has never seen before, then Unicode will
help you to better cope with them.


I prefer Allen working on Tiburon then being jabbed at for fun ;)

--

[Ivan]

To me Q Correll's original post sounded like a joke and the amount of back and forth it generated
added to the fun...

[Q Correll]

Ivan,

| To me Q Correll's original post sounded like a joke

Which was my intent.

[Q Correll]

Thorsten,

Sorry I pushed one of your hot buttons. That was NOT my intent.

I think we can let it drop now.

[TJC Support]

Post by Q Correll
I think we can let it drop now.

Aw, c'mon Q, remember where you are. You can't get off _that_ easy! :^)

Cheers,
Van

[Q Correll]

TJC,

| Aw, c'mon Q, remember where you are. You can't get off that easy! :^)

<chuckle> Ah, yaaass indeedy,... there is that.

[Nick Hodges (Embarcadero)]

Post by Thorsten Engler [NexusDB]
I never said that porting to code from ANSI to Unicode will be without
any issues and gotchas.

Or, put another way, chances are you won't even notice that you are
using Unicode strings. ;-)

[TJC Support]

Post by Nick Hodges (Embarcadero)
Or, put another way, chances are you won't even notice that you are
using Unicode strings. ;-)

Hi Nick,

I think Q and I both learned a lot of lessons from the school of hard
knocks. And while I don't expect any serious problems to crop up due to
your changes in the way things work, the phrase "there's always something"
comes to mind. :^) I expect there will probably be a few issues to crop up
in two areas. 1 is my own sloppy programming. I built my primary
application over 10 years ago and have maintained and added to it over the
years. I've learned a lot in the last 10 years, and if I started the
project over from scratch, it would be a much better piece of code, both
from the architecture/design standpoint and the code quality standpoint. So
there are probably going to be some of those places in the code where I made
bad decisions about string & character handling that'll rear up their ugly
heads. And 2, I use the old Turbopower libraries extensively, and there's
nobody maintaining those now. I did take a quick look at Systools the other
day, and it looks like they were pretty careful about declaring strings as
AnsiString. I was concerned, because a lot of the string handling routines
are written in assembler, but I think they may be okay. But if I have a
_lot_ of problems with those libraries, I'll probably end up having to
abandon them in favor of more up to date stuff that's maintained.

At any rate, I look forward to my first upgrade since D7. The good news for
me is that I don't have a schedule for switching over, so it can take as
long as it takes to work through the code to get it in shape before I switch
to D2009 for production code.

Cheers,
Van Swofford
Tybee Jet Corp.

[Q Correll]

TJC, (I apologize that my simplistic name-capture doesn't come up with
"Van." <g> If you could get your newsreader client to post your signature
with the standard "--" line ahead of "Van Swofford" it might work. <g>)

| I think Q and I both learned a lot of lessons from the school of hard
knocks.

Yep. <g>

| 1 is my own sloppy programming. I built my primary application over 10
years ago and have maintained and added to it over the years. I've
learned a lot in the last 10 years, and if I started the project over from
scratch, it would be a much better piece of code, both from the
architecture/design standpoint and the code quality standpoint. So there
are probably going to be some of those places in the code where I made bad
decisions about string & character handling that'll rear up their ugly
heads.

That's also a ditto for me. ("It just grew and grew and grew..." ;-)

I do think, however, I may have more potential problems with my old
components than my own code. I also use Orpheus4, a TP product as per
your concern #2.

[TJC Support]

Post by Q Correll
TJC, (I apologize that my simplistic name-capture doesn't come up with
"Van." <g> If you could get your newsreader client to post your signature
with the standard "--" line ahead of "Van Swofford" it might work. <g>)

I'm about to switch to Xananews as soon as I get my new machine going,
hopefully in the next week. That _should_ fix it. :^)

Post by Q Correll
That's also a ditto for me. ("It just grew and grew and grew..." ;-)

Hehehe, yep, I know that one.

Post by Q Correll
I do think, however, I may have more potential problems with my old
components than my own code. I also use Orpheus4, a TP product as per
your concern #2.

Yeah, all my UI stuff is Orpheus4, and all my string stuff is Systools. And
"security" is OnGuard. That's in quotes because OG ain't all that secure,
but it's good enough for my audience. And my DB is BTree Filer. You might
say I'm pretty thoroughly TP'd. :^)

Cheers,
Van

[Q Correll]

| I'm about to switch to Xananews as soon as I get my new machine going,
hopefully in the next week. That should fix it. :^)

TJC,

Yes, I think it will. ;-)

| You might say I'm pretty thoroughly TP'd. :^)

Yes I would.

I will be working on converting O4. Perhaps we can exchange notes when
the time comes?

[David Erbas-White]

Post by Q Correll
| I'm about to switch to Xananews as soon as I get my new machine going,
hopefully in the next week. That should fix it. :^)
TJC,
Yes, I think it will. ;-)
| You might say I'm pretty thoroughly TP'd. :^)
Yes I would.
I will be working on converting O4. Perhaps we can exchange notes when
the time comes?

I'd be interested in hearing what other components you use (if
third-party) to replace them. I'm also doing what I can to get away
from using the SysTools package...

David Erbas-White

[David Erbas-White]

Post by Q Correll
I do think, however, I may have more potential problems with my old
components than my own code. I also use Orpheus4, a TP product as per
your concern #2.

Ditto on the Orpheus and SysTools, and AsyncPro is still one of the best
communications libraries around for serial transfers...

David Erbas-White

[David Erbas-White]

Post by willr

Post by Q Correll
Allen,
| So as long as your application continues to get deployed into
| environments that never encounter non-English characters, things should
| just continue to work.
Where can I get a pair of those rose-colored glasses? ;-)

I think they are available at most programming classes -- and "extra
rose coloured" are available at "design" classes.

Yes, but they take them away when you actually complete your first real
'working' program...

David Erbas-White

[Thorsten Engler [NexusDB]]

Post by John Herbster
May I presume like this?
Inc(p);
where MyStr: string; and p: PChar;
And how expensive are these operations during CPU execution?

The Inc(p) used to add 1, now it adds 2 to the pointer. What difference
in performance compared to AnsiString/PAnsiChar do you expect?

--

[Ian Boyd]

Post by Nick Hodges (Embarcadero)

Post by John Herbster
Show us how to iterate through a string of characters with pointers.

Exactly as before -- but don't assume a character is of size 1.

p: Pointer;
p := @MyString[1];
Inc(p);

?

[Nick Hodges (Embarcadero)]

Post by Ian Boyd
p: Pointer;
Inc(p);

That will behave differently, since it (appears) to be assuming the
SizeOf(Char) = SizeOf(Pointer), which is no longer true.

[Thorsten Engler [NexusDB]]

Post by Ian Boyd
p: Pointer;
Inc(p);

You can't do pointer math with untyped pointers. Never worked before.
Not going to start working suddenly. "Pointer" does not know how big
whatever it points to is.

--

[Dave Nottage [TeamB]]

Post by Nick Hodges (Embarcadero)
Exactly as before -- but don't assume a character is of size 1.

Post by John Herbster
Show us how to load and store a string from and to TStreams.

Exactly as before but you can't assume that the length of a string
char is 1.

Post by John Herbster
Show us how to pass strings to and from DLLs.

Just as before, but again, don't assume that Char = 1 byte.

So.. does that mean we should not assume that Char = 1 byte? <g>

[John Herbster]

Nick,

Just a few more questions to help with the book you are writing. <g>

Post by Nick Hodges (Embarcadero)

Post by John Herbster
Show us how to iterate through a string of characters with indexes.

Exactly as before.

What about surrogate pairs? Do you consider such a pair to be two
characters or one?

Post by Nick Hodges (Embarcadero)

Post by John Herbster
Show us how to load and store a string from and to TStreams.

Exactly as before but you can't assume that the length of a string
char is 1.

Does the Length() function return the number of characters or number
of words. Will Length(MyUnicodeString)*SizeOf(MyUniCodeString[1])
give the required number of bytes (not including a terminator) in a
TStream?

Post by Nick Hodges (Embarcadero)

Post by John Herbster
Show us how to replace a character.

Exactly as before.

What if we are replacing a pair with a singleton?

Post by Nick Hodges (Embarcadero)

Post by John Herbster
Show us how to make literal constants an assign them to strings.

Exactly as before.

Can we use the U+ representations in constant statements?
Where do we find the or look up some of the more common U+
character definitions.

Regards, JohnH

[Thorsten Engler [NexusDB]]

John Herbster wrote:

I'm not Nick, but...

Post by John Herbster
What about surrogate pairs? Do you consider such a pair to be two
characters or one?

There is no difference between surrogate pairs in UTF-16 and
leading/trailing bytes you can already encounter in ANSI if the current
codepage happens to be a DBCS/MBCS. If anything surrogate pairs are
much easier to handle because there is no overlap between singelton,
surrogate leading and trailing units.

Post by John Herbster
Does the Length() function return the number of characters or number
of words.

Just the same as with AnsiStrings or dynamic arrays. It returns the
number of elements.

Post by John Herbster
Will Length(MyUnicodeString)*SizeOf(MyUniCodeString[1])
give the required number of bytes (not including a terminator) in a
TStream?

It should, but that code is dangerous as MyUniCodeString[1] would raise
an exception if the string is empty.

Post by John Herbster
What if we are replacing a pair with a singleton?

Absolutely no different from handling DBCS/MBCS in ANSI already.

Post by John Herbster
Can we use the U+ representations in constant statements?
Where do we find the or look up some of the more common U+
character definitions.

You can use the usual # syntax that delphi uses to specify individual
characters by their byte value. Keep in mind that U+ is Hex and # is
decimal.

[Nick Hodges (Embarcadero)]

Post by Thorsten Engler [NexusDB]
I'm not Nick, but...

..you are doing a much better job explaining it than I could. ;-)

[John Herbster]

Post by Thorsten Engler [NexusDB]

Post by John Herbster
What about surrogate pairs? Do you consider such a pair to be two
characters or one?

There is no difference between surrogate pairs in UTF-16 and
leading/trailing bytes you can already encounter in ANSI if the current
codepage happens to be a DBCS/MBCS.

I have never used the DBCS/MBCS code page
-- I think that this may be part of my communication problem.

Post by Thorsten Engler [NexusDB]

Post by John Herbster
Does the Length() function return the number of characters
or number of words.

Just the same as with AnsiStrings or dynamic arrays.
It returns the number of elements.

Is there a standard function returning the number of "characters"?

Post by Thorsten Engler [NexusDB]

Post by John Herbster
Will Length(MyUnicodeString)*SizeOf(MyUniCodeString[1])
give the required number of bytes (not including a terminator) in a
TStream?

It should, but .. MyUniCodeString[1] would raise
an exception if the string is empty.

Of course.

Post by Thorsten Engler [NexusDB]

Post by John Herbster
What if we are replacing a pair with a singleton?

Absolutely no different from handling DBCS/MBCS in ANSI already.

Whatever DBCS/MBCS is. <g>

Post by Thorsten Engler [NexusDB]

Post by John Herbster
Can we use the U+ representations in constant statements?

You can use the usual # syntax that delphi uses to specify individual
characters by their byte value. Keep in mind that U+ is Hex and # is
decimal.

Was that a *no*?

Post by Thorsten Engler [NexusDB]

Post by John Herbster
Where do we find the or look up some of the more common U+
character definitions.

I appreciate your help. Rgds, JohnH

[Thorsten Engler [NexusDB]]

Post by John Herbster
Is there a standard function returning the number of "characters"?

Not that I know of. But it's largely pointless. There are hardly any
cases when you really need to concern yourself with surrogate pairs.

If you really want to count the number of codepoints in an UTF-16
string (but what for?) it's pretty straight forward. Start at the
beginning, any singelton counts as 1. Any leading unit must be followed
by a trailing unit or the string is invalid (same is true in reverse,
any trailing unit must be preceded by a leading unit or the string is
invalid). If you found a leading/trailing pair, count it as 1.

But that doesn't tell you how many glyphs that string would have if
it's displayed. There is no 1:1 relationship between codepoints and
glyphs (the things you recognize as individul elements on when
represented visually).

Thing is, in all cases where you are currently not confronted with
double byte character sets or multi byte character sets with ANSI, you
won't be confronted with surrogate pairs either.

And in the overwhelming majority of cases where you ARE confronted with
DBCS/MBCS in ANSI, you are STILL not going to be confronted with
surrogate pairs in UTF-16.

Post by John Herbster
Whatever DBCS/MBCS is. <g>

That's the thing you have to work with in any but the most trivial ANSI
codepages. Double Byte Character Sets, Multi Byte Character Sets.

Pretty much the same as with surrogate pairs. Just that you can have
more then 1 trailing byte. And the possible values of trailing bytes
can overlap with the values of singeltons. So if you don't always parse
the strings from the beginning and react properly to the leading bytes
to interprete the following trailing bytes differently, you might
totally misinterprete the trailing bytes as singletons with totally
different meaning.

Post by John Herbster

Post by Thorsten Engler [NexusDB]

Post by John Herbster
Can we use the U+ representations in constant statements?

You can use the usual # syntax that delphi uses to specify
individual characters by their byte value. Keep in mind that U+ is
Hex and # is decimal.

Was that a no?

That was a "you can use what you've always used" in Delphi.

Post by John Herbster

Post by Thorsten Engler [NexusDB]

Post by John Herbster
Where do we find the or look up some of the more common U+
character definitions.

Start > Accessories > System Tools > Character Map. Knock yourself out
(pay close attention to the status bar).


--

[John Herbster]

Post by Thorsten Engler [NexusDB]

Post by John Herbster
Is there a standard function returning the number of "characters"?

Not that I know of. But it's largely pointless. There are hardly any
cases when you really need to concern yourself with surrogate pairs.
If you really want to count the number of codepoints in an UTF-16
...
But that doesn't tell you how many glyphs that string would have if
it's displayed. There is no 1:1 relationship between codepoints and
glyphs (the things you recognize as individul elements on when
represented visually).

Oh -- More definition problems:
What is a "character", if it exists, in this brave new world?
What is a "glyph"?
What is a "codepoint"?

Post by Thorsten Engler [NexusDB]
Thing is, in all cases where you are currently not confronted with
double byte character sets or multi byte character sets with ANSI,
you won't be confronted with surrogate pairs either.
And in the overwhelming majority of cases where you ARE confronted with
DBCS/MBCS in ANSI, you are STILL not going to be confronted with
surrogate pairs in UTF-16.

Post by John Herbster
Whatever DBCS/MBCS is. <g>

That's the thing you have to work with in any but the most trivial ANSI
codepages. Double Byte Character Sets, Multi Byte Character Sets.

Thank goodness for trivial!

Post by Thorsten Engler [NexusDB]
...

Post by John Herbster

Post by Thorsten Engler [NexusDB]

Post by John Herbster
Can we use the U+ representations in constant statements?

You can use the usual # syntax that delphi uses to specify
individual characters by their byte value. Keep in mind that U+ is
Hex and # is decimal.

Was that a no?

That was a "you can use what you've always used" in Delphi.

I conclude for now, that one does not ever need to use "U+"
representations in consts.

Post by Thorsten Engler [NexusDB]

Post by John Herbster

Post by Thorsten Engler [NexusDB]

Post by John Herbster
Where do we find the or look up some of the more common U+
character definitions.

Start > Accessories > System Tools > Character Map.
(pay close attention to the status bar).

Thorsten, Thank you.

I hope that Unicode will help us to communicate better. Will
Unicode do anything for decimal points and thousand separators?
Will it help formatting messages in newsgroup readers?

Regards, JohnH

[Thorsten Engler [NexusDB]]

Post by John Herbster
What is a "character", if it exists, in this brave new world?
What is a "glyph"?
What is a "codepoint"?

To quote myself:

I would like to strongly recommend that everyone heads over to

http://www.unicode.org/versions/Unicode5.1.0/

And read the Introduction:

http://www.unicode.org/versions/Unicode5.0.0/ch01.pdf

and at least fly over the General Structure:

http://www.unicode.org/versions/Unicode5.0.0/ch02.pdf

These 2 chapters together should answer 99% of any question someone
here might have about Unicode.

Post by John Herbster
Thank goodness for trivial!

If you have no intention of supporting anything that you didn't support
so far with using just an english ANSI codepage then unicode is
absolute trivial for you. As far as you concerned the only difference
is basically going to be that chars are now two bytes instead of one
and the high byte is basically always going to be 0 for you.

Even if you want to support any of the complexer scripts, you usually
don't have to concern yourself with any of the details of unicode. The
user enters a string, store it the way it is. You want to display it?
Hand it to the Windows API the way it is. Simple. You want to compare 2
strings? Pass them to the CompareStringW API.

There are a few windows APIs that you might not have used before that
can be helpful:

NormalizeString:
http://msdn.microsoft.com/en-us/library/ms776395(VS.85).aspx

FoldString:
http://msdn.microsoft.com/en-us/library/cc709430(VS.85).aspx

LCMapString/LCMapStringEx:
http://msdn.microsoft.com/en-us/library/ms776290(VS.85).aspx
http://msdn.microsoft.com/en-us/library/ms776387(VS.85).aspx

The parameters might look a bit scary, but I'm pretty sure that Tiburon
will come with some form of nicer wrappers for such things.

Post by John Herbster
I conclude for now, that one does not ever need to use "U+"
representations in consts.

Use #$ instead of U+

--

[Jens Mühlenhoff]

Post by Thorsten Engler [NexusDB]

Post by John Herbster

Post by Thorsten Engler [NexusDB]

Post by John Herbster
Can we use the U+ representations in constant statements?

You can use the usual # syntax that delphi uses to specify
individual characters by their byte value. Keep in mind that U+ is
Hex and # is decimal.

Was that a no?

That was a "you can use what you've always used" in Delphi.

To put it another way, #$1A3B (hexadecimal character) is the delphi
representation of U+1A3B. The "U+" notation would really be superflous here.

[Chad Z. Hower aka Kudzu]

John, the concept of what is or is not a "character" is very much
language-specific. So, when asking such questions, it is good to be
very specific about what problem you are trying to solve.

For example - Chinese which a character is a word. And believe it or not,
there are more complex scenarios in other languages...

[Thorsten Engler [NexusDB]]

Post by Chad Z. Hower aka Kudzu
For example - Chinese which a character is a word. And believe it or
not, there are more complex scenarios in other languages...

Actually, you are refering to visual representation here. That's a
Glyph, it's rather common in Unicode that multiple codepoints together
contribute to the visual representation of a single glyph (but again,
cases where you are not confronted with DBCS/MBCS when using ANSI you
are not going to be confronted with this in Unicode either).

I would like to strongly recommend that everyone heads over to

http://www.unicode.org/versions/Unicode5.1.0/

And read the Introduction:

http://www.unicode.org/versions/Unicode5.0.0/ch01.pdf

and at least fly over the General Structure:

http://www.unicode.org/versions/Unicode5.0.0/ch02.pdf

These 2 chapters together should answer 99% of any question someone
here might have about Unicode.


--

[Chad Z. Hower aka Kudzu]

Post by Thorsten Engler [NexusDB]
Actually, you are refering to visual representation here. That's a
Glyph, it's rather common in Unicode that multiple codepoints together
contribute to the visual representation of a single glyph (but again,
cases where you are not confronted with DBCS/MBCS when using ANSI you
are not going to be confronted with this in Unicode either).

I know thats the case for Simplified, is it the same for traditional?

[Thorsten Engler [NexusDB]]

Post by Chad Z. Hower aka Kudzu
I know thats the case for Simplified, is it the same for traditional?

I wasn't specifically talking about Traditional and Simplified Chinese
there but Unicode and complex scripts in general.

There exists about 70000 codepoints for fully composed Han Ideographs.
But there are also mechanisms for decompoising them into individual
components (up to 16 can make up a single composed Ideograph) as well
as many more Ideographs which can only be described in a decomposed
fashion using Ideographic Description Sequences.

Both Simplified and Traditional Chinese (as well as Japanese, Korean
and to some limit Vietnamese) take their visual representations from
these Han Ideographs, with many Ideographs used most of these languages
(but not always with the same meaning).

As far as Unicode is concerned there are no specific traditional or
simplified characters. Most Ideographs used in Simplified Chinese are
also used in Traditional Chinese (There are some Ideographs used in
Simplified which are, well, simplified, versions of Traditional
Ideographs, but mostly the simplification is based on using a single
Ideograph to represent the meaning of different Traditional Ideographs).

There are a lot more different Ideographs used in Traditional Chinese,
so it's more often in Traditional then Simplified that no precomposed
codepoint exist and a single Ideograph needs to be represented with a
Ideographic Description Sequence.

Cheers,
Thorsten

--

[Craig Stuntz [TeamB]]

Post by John Herbster
Is there a standard function returning the number of "characters"?

John, the concept of what is or is not a "character" is very much
language-specific. So, when asking such questions, it is good to be
very specific about what problem you are trying to solve.

[Remy Lebeau (TeamB)]

Post by John Herbster
Whatever DBCS/MBCS is. <g>

MBCS = Multi-Byte Character Set

DBCS = Double-Byte Character Set.


Gambit

[Remy Lebeau (TeamB)]

Post by John Herbster
What about surrogate pairs? Do you consider such a pair
to be two characters or one?

When indexing through a String, they are separate *physical* characters -
they are each stored in a separate WideChar, and can thus be directly
indexed individually. To retreive a *logical* character (aka a Unicode
codepoint), then they have to be interpretted together. This is no
different than using MBCS or DBCS in AnsiString.

Post by John Herbster
Does the Length() function return the number of characters
or number of words.

The meaning of Length() has not changed. It has always returned the number
of *physical* entries, not the number of *logical* entities. That is the
same whether it s used with an AnsiString, UnicodeString, dynamic array,
etc. In the case of a UnicodeString, that means the number of physically
allocated WideChar entries, not the number of encoded Unicode codepoints.

Post by John Herbster
Will Length(MyUnicodeString)*SizeOf(MyUniCodeString[1]) give
the required number of bytes (not including a terminator) in a TStream?

Yes.

Post by John Herbster

Post by John Herbster
Show us how to replace a character.

Exactly as before.

What if we are replacing a pair with a singleton?

Then you would have to pull the string apart and put it back together
manually - the same as you would have had to do with MBCS and DBCS in
AnsiString. For example:

if IsHighSurrogate(TheString[Index]) then
TheString := Copy(TheString, 1, Index-1) + TheSingleton +
Copy(TheString, Index+2, MaxInt)
else if IsLowSurrogate(TheString[Index]) then
TheString := Copy(TheString, 1, Index-2) + TheSingleton +
Copy(TheString, Index+1, MaxInt);
else
TheString[Index] := TheSinglegon;

Post by John Herbster
Can we use the U+ representations in constant statements?

I am not sure, but I think UnicodeString constants have to be UTF-16
compliant. Unless you specify them in UCS4/UTF-32 and let the compiler
convert for you at runtime.

Post by John Herbster
Where do we find the or look up some of the more common U+ character
definitions.

The Unicode standard.


Gambit

[Lucian Radulescu]

If I have old code like:

NameRecType = packed record
Gender : Byte;
SurName : String[70];
FirName : String[20];
MidNames: String[30];
DOB : TDateTime;
end;

and I blockread/blockwrite vars of NameRecType from untyped files, do I
need to change anything or it will just compile and I don't have to
worry about it?

Lucian

[Thorsten Engler [NexusDB]]

Post by Lucian Radulescu
NameRecType = packed record
Gender : Byte;
SurName : String[70];
FirName : String[20];
MidNames: String[30];
DOB : TDateTime;
end;
and I blockread/blockwrite vars of NameRecType from untyped files, do
I need to change anything or it will just compile and I don't have to
worry about it?

The shortstring type has not changed in any way.

--

[Serge Dosyukov (Dragon Soft)]

1) Few functions are expecting PAnsiChar/PWideChar instead of
AnsiCar/WideChar (windows API)
2) working with Windows API, be aware of what you are passing around
(windows messages)
3) use Length()
4) P-strings are still #0 terminated, but instead of #00, you might see
#0000.

In Delphi 7
var
LC: char;
LC2: widechar;
LC3: ansichar;
begin
ShowMessage(IntToStr(SizeOf(LC)) + ', ' + IntToStr(SizeOf(LC2)) + ', ' +
IntToStr(SizeOf(LC3)));
end;

gives "1, 2, 1"

where now you may get

gives "2, 2, 1"

Post by Lee Jenkins
Has anyone posted information concerning do's and dont's for
Unicode support in upcoming Delphi versions?