https://wortschatz.uni-leipzig.de/en/download/hindi
hin_mixed_2019_30K
hin-in_web_2015_30K
hin_news_2011_10K
hin_newscrawl_2016_10K
hin_newscrawl_2017_10K
hin_wikipedia_2016_10K
---
total: 100K
---
For conversion, multiple steps are necessary:
1. Aksharamukha conversion tool, preferably the python script. Conversion from devanagiri with schwa deletion to Roman IAST.
2. Conversion script to replace ā|æ|ç|ḍ|ĕ|è|ġ|ī|ï|ṃ|ṇ|ṉ|ṅ|Ñ|Ó|ô|ŏ|ṛ|ṟ|ṝ|ś|ṣ|ṭ|ū|ẏ with normal letters.
3. sed-command in terminal (linux)
...
1. Aksharamukha conversion tool (source: https://aksharamukha.appspot.com/):
Code: Select all
#!/usr/bin/python3
# coding: utf-8
import sys
from aksharamukha import transliterate
import argparse
def transliterateRoman(s):
return transliterate.process('Devanagari', 'IAST', s, pre_options=['RemoveSchwaHindi'])
def batch_transliterate(dev_file, rr_file):
for line in dev_file:
rr_file.writelines([transliterateRoman(line)])
return 0
if __name__ == '__main__':
ap = argparse.ArgumentParser()
ap.add_argument('devanagari_file', nargs='?', type=argparse.FileType('r'), default=sys.stdin)
ap.add_argument('romanIAST_file', nargs='?', type=argparse.FileType('w'), default=sys.stdout)
args = ap.parse_args()
exit(batch_transliterate(args.devanagari_file, args.romanIAST_file))
run script with:
Code: Select all
python3 convert_hindi_roman.py hindi2020.txt hindiromanIAST2020.txt2. Conversion script to replace (ā|æ|ç|ḍ|ĕ|è|ġ|ḥ|ī|ï|ṃ|ṇ|ṉ|ṅ|ñ|Ñ|Ó|ô|ŏ|ṛ|ṟ|ṝ|ś|ṣ|ṭ|ū|ü|ẏ) with normal letters:
Code: Select all
$rules = New-Object System.Collections.Hashtable
$rules.ā = "a"
$rules.æ = "ae"
$rules.ç = "c"
$rules.ḍ = "d"
$rules.ĕ = "e"
$rules.è = "e"
$rules.ġ = "g"
$rules.ḥ = "h"
$rules.ī = "i"
$rules.ï = "i"
$rules.ṃ = "m"
$rules.ṇ = "n"
$rules.ṉ = "n"
$rules.ṅ = "n"
$rules.ñ = "n"
$rules.Ñ = "n"
$rules.Ó = "O"
$rules.ô = "o"
$rules.ŏ = "o"
$rules.ṛ = "r"
$rules.ṟ = "r"
$rules.ṝ = "r"
$rules.ś = "s"
$rules.ṣ = "s"
$rules.ṭ = "t"
$rules.ū = "u"
$rules.ü = "u"
$rules.ẏ = "y"
$file = Get-Content -Path $args[0]
$a = [regex]'(ā|æ|ç|ḍ|ĕ|è|ġ|ḥ|ī|ï|ṃ|ṇ|ṉ|ṅ|ñ|Ñ|Ó|ô|ŏ|ṛ|ṟ|ṝ|ś|ṣ|ṭ|ū|ü|ẏ)‘
$a_cb = {$rules[$args[0].Groups[1].Value] + $args[0].Groups[2].Value}
Write-Progress -Activity "a" -PercentComplete 4
$file = $a.Replace($file, $a_cb)
Set-Content -Path "$args.converted.txt" -Value $file
echo "Finished"
in terminal run:
Code: Select all
pwshCode: Select all
./IASTtoclear.ps1 hindiromanIAST2020.txt---
3. sed-command in terminal (linux) to remove some diacritics, that can not be remove with the simple conversion script from step 2 (there still is one diacritic sign I was not able to replace, it looks like [_]:
sed -i 's/̤//g' hindiromanIASTconverted2020.txt
sed -i 's/̐//g' hindiromanIASTconverted2020.txt
sed -i 's/̈//g' hindiromanIASTconverted2020.txt
---
Letter frequency chart (all alphabetic characters + punctuation characters with >0.1% frequency):
- hindi-roman-IAST-characterfrequency-with-punctuation.png (143.92 KiB) Viewed 37300 times
- 100HINDI.png (195.16 KiB) Viewed 37297 times
- HIN50ENG50.png (191.71 KiB) Viewed 37297 times
- ENG100hindiroman.png (195.22 KiB) Viewed 37297 times