Arabic News

Arabisch-News

Arabische Nachrichten für die Sprachmodellierung von gesammelt von

• BBC Arabisch
• Euronews
• Aljazeera
• CNN Arabisch
• RT -Arabisch

Diese Nachrichten werden von News-bitte Python Library gesammelt

Neuigkeiten und Titel python json2corpus.py extrahieren

Crawl-Datum: 19-04-2019

---

Korpusinformationen

Korpus	Größe	Anzahl der Wörter
Schlagzeilen	54 m	487674
JSC	395 m	1525372
Rt	713 m	3411451
CNN	94 m	317260
BBC	854 m	1764796
Euronews	279m	517227

---

Protokoll:

 corpus name: 09/arabic.euronews.com/
processing /home/motaz/newsgitrepo/data/2019/04/09/arabic.euronews.com/
# of files 46468
46079 is done out of 46468
number of files 46468
{'ar': 46468}
short count 389
.json
number of num_parts 18000
len of sub_lists 3
.txt
number of num_parts 18000
len of sub_lists 3
-------------------------------
corpus name: 09/bbc.com/
processing /home/motaz/newsgitrepo/data/2019/04/09/bbc.com/
# of files 212271
94734 is done out of 212271
number of files 212271
{'pt': 1, 'ar': 97468, 'fa': 114648, 'en': 154}
short count 2734
.json
number of num_parts 18000
len of sub_lists 6
.txt
number of num_parts 18000
len of sub_lists 6
-------------------------------
corpus name: 19/aljazeera.net/
processing /home/motaz/newsgitrepo/data/2019/04/19/aljazeera.net/
# of files 249106
109141 is done out of 249106
number of files 249106
{'ar': 170003, 'en': 3}
short count 60862
.json
number of num_parts 18000
len of sub_lists 7
.txt
number of num_parts 18000
len of sub_lists 7
-------------------------------
corpus name: 19/arabic.rt.com/
processing /home/motaz/newsgitrepo/data/2019/04/19/arabic.rt.com/
# of files 368920
334268 is done out of 368920
number of files 368920
{'ar': 368857}
short count 34589
.json
number of num_parts 18000
len of sub_lists 19
.txt
number of num_parts 18000
len of sub_lists 19
-------------------------------
corpus name: 19/arabic.cnn.com/
processing /home/motaz/newsgitrepo/data/2019/04/19/arabic.cnn.com/
# of files 30338
30140 is done out of 30338
number of files 30338
{'ar': 30338}
short count 198
.json
number of num_parts 18000
len of sub_lists 2
.txt
number of num_parts 18000
len of sub_lists 2
-------------------------------
all done

---

Komprimieren Sie Dateien in einem Verzeichnis

 $ tree dir1/
dir1/
|-- dir11
|   |-- file11
|   |-- file12
|   `-- file13
|-- file1
|-- file2
`-- file3

Führen Sie nun den Gzip -Befehl aus

$ gzip -r dir1

nach

 $ tree dir1/
dir1/
|-- dir11
|   |-- file11.gz
|   |-- file12.gz
|   `-- file13.gz
|-- file1.gz
|-- file2.gz
`-- file3.gz

Löschen Sie viele Dateien

find . -name '*.html.gz' -print0 | xargs -0 rm

Komprimieren Sie Dateien mit 7z