When working with non-US customers, users often have characters in their names like \u00eb, \u00f3, \u00e7 and so on. Most of the time, a ‘human process’ converts these to their simple equivalent of e, o and c for use in computerized systems.<\/p>\n\n\n\n
When searching for such a mapping of special characters to ‘safe’ characters I had a hard time finding a good list or PowerShell method to automatically convert special characters to standard A-Z characters so I wrote one:<\/p>\n\n\n
\nfunction get-sanitizedUTF8Input{\n Param(\n [String]$inputString\n )\n $replaceTable = @{\"\u00df\"=\"ss\";\"\u00e0\"=\"a\";\"\u00e1\"=\"a\";\"\u00e2\"=\"a\";\"\u00e3\"=\"a\";\"\u00e4\"=\"a\";\"\u00e5\"=\"a\";\"\u00e6\"=\"ae\";\"\u00e7\"=\"c\";\"\u00e8\"=\"e\";\"\u00e9\"=\"e\";\"\u00ea\"=\"e\";\"\u00eb\"=\"e\";\"\u00ec\"=\"i\";\"\u00ed\"=\"i\";\"\u00ee\"=\"i\";\"\u00ef\"=\"i\";\"\u00f0\"=\"d\";\"\u00f1\"=\"n\";\"\u00f2\"=\"o\";\"\u00f3\"=\"o\";\"\u00f4\"=\"o\";\"\u00f5\"=\"o\";\"\u00f6\"=\"o\";\"\u00f8\"=\"o\";\"\u00f9\"=\"u\";\"\u00fa\"=\"u\";\"\u00fb\"=\"u\";\"\u00fc\"=\"u\";\"\u00fd\"=\"y\";\"\u00fe\"=\"p\";\"\u00ff\"=\"y\"}\n\n foreach($key in $replaceTable.Keys){\n $inputString = $inputString -Replace($key,$replaceTable.$key)\n }\n $inputString = $inputString -replace '[^a-zA-Z0-9]', ''\n return $inputString\n}\n\n#example usage:\nget-sanitizedUTF8Input -inputString \"J\u00f6s\u00e8\"\n#result:\nJose\n<\/pre><\/div>\n\n\nEdit: my colleague Gerbrand<\/a> alerted me to a post by Gr\u00e9gory Schiro<\/a> which solves this issue much more elegantly using native .NET functions. My slightly modified version to really ensure nothing non a-zA-Z0-9 gets past the function:<\/p>\n\n\n\nfunction Remove-DiacriticsAndSpaces\n{\n Param(\n [String]$inputString\n )\n $objD = $inputString.Normalize([Text.NormalizationForm]::FormD)\n $sb = New-Object Text.StringBuilder\n \n for ($i = 0; $i -lt $objD.Length; $i++) {\n $c = [Globalization.CharUnicodeInfo]::GetUnicodeCategory($objD[$i])\n if($c -ne [Globalization.UnicodeCategory]::NonSpacingMark) {\n [void]$sb.Append($objD[$i])\n }\n }\n \n $sb = $sb.ToString().Normalize([Text.NormalizationForm]::FormC)\n return($sb -replace '[^a-zA-Z0-9]', '')\n}\n#example usage:\nRemove-DiacriticsAndSpaces -inputString \"J\u00f6s\u00e8\"\n#result:\nJose\n\n<\/pre><\/div>\n\n\nAnd an even easier oneliner I converted to a function by John Seerden<\/a>:<\/p>\n\n\n\nfunction Remove-DiacriticsAndSpaces\n{\n Param(\n [String]$inputString\n )\n #replace diacritics\n $sb = [Text.Encoding]::ASCII.GetString([Text.Encoding]::GetEncoding(\"Cyrillic\").GetBytes($inputString))\n\n #remove spaces and anything the above function may have missed\n return($sb -replace '[^a-zA-Z0-9]', '')\n}\n<\/pre><\/div>\n\n\nAnd the most advanced function I’ve found so far is by
Daniele Catanesi (PsCustomObject)<\/a>: https:\/\/github.com\/PsCustomObject\/New-StringConversion\/blob\/master\/New-StringConversion.ps1<\/a> in which all features of above functions are supported and parameterized.<\/p>\n\n\n\n<\/p>\n\n\n\n
<\/p>\n","protected":false},"excerpt":{"rendered":"
When working with non-US customers, users often have characters in their names like \u00eb, \u00f3, \u00e7 and so on. Most of the time, a ‘human process’ converts these to their simple equivalent of e, o and c for use in computerized systems. When searching for such a mapping of special characters to ‘safe’ characters I … Continue reading Removing special characters from UTF8 input for use in email addresses or login names<\/span>