<!doctype HTML public "-//W3C//DTD HTML 4.0 Frameset//EN">

<html>

<!--(==============================================================)-->
<!--(Document created with RoboEditor. )============================-->
<!--(==============================================================)-->

<head>

<title>Working_with_Unicode</title>

<!--(Meta)==========================================================-->

<meta http-equiv=content-type content="text/html; charset=windows-1252">
<meta name=generator content="RoboHELP by eHelp Corporation - www.ehelp.com">
<meta name=generator-major-version content=0.1>
<meta name=generator-minor-version content=1>
<meta name=filetype content=kadov>
<meta name=filetype-version content=1>
<meta name=page-count content=1>
<meta name=layout-height content=892>
<meta name=layout-width content=1024>


<!--(Links)=========================================================-->

<link rel="StyleSheet" href="default.css">



</head>

<!--(Body)==========================================================-->


<body>

<h1>Working with Unicode</h1>

<p>Starting with VisualText 2.0.1, a Unicode VisualText executable is available 
 with every release.</p>

<p>Unicode is well-documented on the web. <!--kadov_tag{{<spaces>}}-->&nbsp;<!--kadov_tag{{</spaces>}}-->Basically, 
 Unicode is a character set that supports the major languages and character 
 sets. <!--kadov_tag{{<spaces>}}-->&nbsp;<!--kadov_tag{{</spaces>}}-->To 
 do this, characters consist of two bytes, rather than the ANSI or ASCII 
 representation of characters with one byte.</p>

<p>While the advantages of Unicode are tremendous, it also adds substantial 
 complexity and headaches to programs that use it. <!--kadov_tag{{<spaces>}}-->&nbsp;<!--kadov_tag{{</spaces>}}-->Also, 
 Unicode is necessarily processed more slowly than single-byte character 
 representations.</p>

<p>VisualText support for Unicode is an ongoing enterprise. <!--kadov_tag{{<spaces>}}-->&nbsp;<!--kadov_tag{{</spaces>}}-->The 
 2.0.1 release provides default tokenization for the world's character 
 sets (or &quot;scripts&quot;), as well as supporting Unicode characters 
 in NLP++ strings, in the GUI, in knowledge bases, compiled analyzers, 
 and the like.</p>

<p>In subsequent releases, we plan to add NLP++ functions that will help 
 characterize and manipulate Unicode constructs in text, parse trees, strings, 
 and the knowledge base.</p>

<h2>Windows and Unicode</h2>

<p>To display and input Unicode fonts, you'll need to consult your Microsoft 
 Windows documentation. <!--kadov_tag{{<spaces>}}-->&nbsp;<!--kadov_tag{{</spaces>}}-->For 
 example, in Windows XP, select</p>

<p>Start Button &gt; Settings &gt; Control Panel &gt; Regional and Language 
 Settings</p>

<p>The Languages tab there lets you install fonts for complex and East 
 Asian character sets. <!--kadov_tag{{<spaces>}}-->&nbsp;<!--kadov_tag{{</spaces>}}-->The 
 Details button within the Languages tab lets you set up input in various 
 character sets, and so on.</p>

<h2>NOTES</h2>

<p>[1] If using English only or small Western alphabets (e.g., French, 
 Spanish, German), consider staying with ANSI VisualText.</p>

<p>[2] An analyzer should be used with Unicode or ANSI VisualText, but 
 not both.</p>

<p>[3] Unicode VisualText writes files out using the UTF-8 format and expects 
 to read files in the same format. <!--kadov_tag{{<spaces>}}-->&nbsp;<!--kadov_tag{{</spaces>}}-->If 
 an input file contains Unicode characters, then VisualText expects it 
 to have the Byte Order Marker (BOM) as required of Windows files. <!--kadov_tag{{<spaces>}}-->&nbsp;<!--kadov_tag{{</spaces>}}-->A 
 nice thing about UTF-8 is that normal ANSI or ASCII files are automatically 
 in the right format, with or without a BOM header.</p>

<p><img src="Tutorials/GO.gif"
		style="width: 19px;
				height: 16px;
				border-style: none;
				margin-top: 0px;
				margin-bottom: 0px;
				margin-left: 0px;
				margin-right: 0px;"
		width=19
		height=16
		border=0> If a file does not appear to be read correctly by VisualText, 
 then it may have come from a non-Windows source. <!--kadov_tag{{<spaces>}}-->&nbsp;<!--kadov_tag{{</spaces>}}-->Try 
 opening the file in an application such as WordPad, add a character, then 
 remove the character, then save the file as UTF-8. <!--kadov_tag{{<spaces>}}-->&nbsp;<!--kadov_tag{{</spaces>}}-->This 
 should automatically add the BOM head to the beginning of the file.</p>

<p>[4] KB dictionary indexing. <!--kadov_tag{{<spaces>}}-->&nbsp;<!--kadov_tag{{</spaces>}}-->We'll 
 be providing analyzer templates specific for Unicode. <!--kadov_tag{{<spaces>}}-->&nbsp;<!--kadov_tag{{</spaces>}}-->You 
 can also manually add dictionary indexes to the knowledge base (by editing 
 the <span style="font-weight: bold;">kb\user\hier.kb</span> file). <!--kadov_tag{{<spaces>}}-->&nbsp;<!--kadov_tag{{</spaces>}}-->Contact 
 us at support@textanalysis.com for help with these issues.</p>

<p>add hier &quot;concept&quot; &quot;sys&quot; &quot;dict&quot; &quot;a&quot; 
 &quot;z&quot; &quot;zy&quot;<br>
add hier &quot;concept&quot; &quot;sys&quot; &quot;dict&quot; &quot;a&quot; 
 &quot;&#12354;&quot;<br>
add hier &quot;concept&quot; &quot;sys&quot; &quot;dict&quot; &quot;a&quot; 
 &quot;&#12354;&quot; &quot;&#12354;&quot;</p>

<p>Basically, you'll want to add lines like the second and third above, 
 in order to have Unicode characters distributed well within the KB dictionary.</p>

<p>[5] No particular updates have been made yet to RUG (Automated Rule 
 Generation) for Unicode VisualText.</p>

<p>&nbsp;</p>

</body>

</html>
