jasterv/bleu-score.py
1
0
Fork 0
mirror of https://codeberg.org/JasterV/bleu-score.py.git synced 2026-04-26 18:40:02 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions

First commit

Browse source
This commit is contained in:
Víctor Martínez 2020-04-21 02:11:50 +02:00
parent a4151ba767
commit d802cf87bf
3 changed files with 304 additions and 0 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

104
src/bleu_score.py Normal file
View file

@ -0,0 +1,104 @@
#!/usr/bin/env python3
from sys import argv, exit
import math
import re
import string
def read_file(filepath):
"""Reads a file line by line
Arguments: filepath -> String
Return: List of Strings
"""
with open(filepath, "r") as file:
return list(map(lambda s: s.strip(), file.readlines()))
def tokenize(s):
"""Tokenize a string
Arguments: s -> String
Return: A list of words
"""
_splited = s.split()
regex = re.compile(f"[{re.escape(string.punctuation)}]")
return list(map(lambda str: regex.sub("", str), _splited))
def n_grama(sentence, n):
"""Returns a words n_gram from a sentence
Arguments:
sentence -> String
n -> int
"""
t = tokenize(sentence)
n_grams = zip(*[t[i:] for i in range(n)])
return list(map(lambda n_gram: ' '.join(n_gram), n_grams))
def calculate_bp_penality(candidate, reference):
""" Calculates the Brevity Penalty for 2 sentences
Arguments:
candidate -> String
reference -> String
"""
c, r = len(tokenize(candidate)), len(tokenize(reference))
return 1 if c > r else math.exp(1-(r/c))
def calculate_p(candidate, reference):
"""Calculates the relation between two n_grams
Arguments:
candidate -> List of tuples (n-grams)
reference -> List of tuples (n-grams)
"""
matches = 0
for grama in candidate:
if grama in reference:
matches += 1
return matches/len(candidate)
def calculate_bleu(candidate, reference):
"""Calculates the bleu score for 2 sentences
Arguments:
candidate -> String
reference -> String
"""
pn_sum = 0
for n in range(1, 4):
cand_grama, ref_grama = n_grama(candidate, n), n_grama(reference, n)
p = calculate_p(cand_grama, ref_grama)
if p != 0:
# If there are no matches between n_grams,
# don't add nothing
pn_sum += (0.33 * math.log(p))
bp = calculate_bp_penality(candidate, reference)
return bp * math.exp(pn_sum)
def bleu_score(output, reference):
"""Calculates and prints each bleu score for each sentence on the texts
Arguments:
output -> List of Strings (List of sentences)
reference -> List of Strings (List of sentences)
"""
num_sentences = min(len(output), len(reference))
for i in range(num_sentences):
if i == 5:
print("Do you want to calculate all sentences? (y/n)")
answer = input()
if answer.lower() == 'n':
break
out_sentence, ref_sentence = output[i], reference[i]
bleu_score = calculate_bleu(out_sentence, ref_sentence)
print(
f"Sentence {i + 1}, Score: {bleu_score}")
if __name__ == "__main__":
if len(argv) != 3:
print("bleu_score.py <output file> <reference file>")
exit()
out_path, ref_path = argv[1], argv[2]
output, reference = read_file(out_path), read_file(ref_path)
bleu_score(output, reference)

100
src/out.txt Normal file
View file

@ -0,0 +1,100 @@
Munich 1856: four maps that change your view of the city
An insane institution where young people are supposed to meet today.
A Gruftkapelle, where the S-Bahn tunnel is now being dug.
Small gardeners manage the former land of farmers.
The oldest official map of Munich reveals exciting stories.
It is annoying when maps are not up to date.
Everyone knows this, because the car navigation system showed a green meadow instead of a bypass.
The historical maps of the digital Bayern-Atlas, an offer of the Geoportals Bayern of the state government, are anything but current - but for this very reason very revealing.
Especially if you compare them with current online maps.
Then it becomes clear how cities and municipalities in the distribution area of Munich Mercury have changed since the 19th century.
The digital Bayern-Atlas is a patchwork that is not immediately recognizable as such.
More detailed maps, so-called primordial maps, have been incorporated into the large overview map for cities and municipalities.
According to the Bayerische Vermessungsverwaltung (Bayerische Vermessungsverwaltung), these were created between 1810 and 1864 and were the result of Bavaria's first survey.
Her client: Napoleon.
The French army leader occupied Bavaria since the spring of 1800 during the Second Coalition War with Austria.
Napoleon wanted an overview of the occupied country and commissioned a topographic recording of Bavaria - for military purposes, of course.
Even though the French troops finally departed with the peace of Lunéville on February 9, 1801, it was the present neighbours who created the idea of a nationwide map of Bavaria.
Elector Max IV. Joseph continued to drive the project forward, founding the "topographic Bureau" - and with it the Bayerische Vermessungsverwaltung.
The original manuscript for Munich dates back to 1856 and, like all others, can be found on the website of the Bayerische Landesbibliothek.
The comparison between past and present is most impressive, using the example of the state capital.
The city has expanded in all directions and has grown together.
An example of this is Untersendling, which appears relatively isolated in the map of 1856.
Sendling was incorporated as a district of Munich 21 years after the map was drawn up, i.e. in 1877.
Today, Untersendling is in the middle of the city.
The houses of Untersendling are concentrated around 1856 along what is now Plinganserstrasse, which was only named in 1878 after Georg Sebastian Plinganser, a leader of the uprising of the Bavarian peasants against Austria in 1705.
Where the Westpark is today, you can see nothing but fields - at first glance: when zooming in, a smaller map with dozens of lines and numbers appears: handwritten numbered flurries from the so-called "original photograph," from which the property tax was derived.
Small plots no longer occupy farmers there - but small gardeners of the "land in sun" plant.
A large green area is emerging north of Untersendling.
The Theresienwiese, the site of the Oktoberfest since 1810, was in fact still a meadow in the 19th century - and not a concrete desert.
On the other side of the Isar, the historical map gives an insight into the history of psychiatric medicine.
In the original position sheet, in the area of today's Au-Haidhausen district, one reads not only the street name "am Gasteig" or from the hill "auf der Lüften" - but also in large letters "erren Anstalt."
Yes, people with delusions were once locked up here on Auerfeldstrasse.
The Munich Kreis-Irrenanstalt "auf der Auer Lüften" was founded there in 1858/59.
From 1872, Dr. Bernhard von Gudden was in charge of the institution - the psychiatrist brought in connection with the death of King Ludwig II.
"He was progressive for his time and ended, for example, the chaining of the so-called irons.
He also prohibited the staff from using violence against the inmates, "says Hermann Wilhelm, head of the Haidhausen Museum, who has been working with the institution for an exhibition.
At first there was room for 280 patients, later it was expanded to 500 places.
At some point, this capacity was no longer sufficient - the reason for the construction of the Kreisirrenanstalt in Haar in 1905.
Since 2006, the Isar-Amper-Klinikum München-Ost, one of the largest psychiatrists in Germany, has been located there.
Young people are now meeting in Munich's Salesianum on the site of the former Auer Anstalt.
The Order acquired the building complex in 1920 and converted it into a youth home with training facilities.
To the west lies the football pitch of the TSV München Ost, to the east residential buildings from the 1980s.
In 2015, the children's house "Casa Don Bosco" was built - in architectural reference to the original building of the Irrenanstalt, which was destroyed in bombing raids during the Second World War.
The Marienplatz was listed as "Haupt Platz" around 1856, the Kaufingerstrasse was called "Kaufinger Gasse."
The Second World War also brought a decisive change in the heart of Munich.
The Allied bombs tore many holes in the city centre in December 1944.
A big one remained to this day - at the Marienhof.
The English Institute, small shops and a chapel from the Middle Ages once stood at the Marienhof, where preparatory work is being carried out these days and the ground-breaking ceremony for the second regular S-Bahn line will take place on 5 April 2017.
With the slider, the old Munich map can be compared to the aerial view of today - and the hole at the Marienhof can be filled up.
Civil rights group issues travel warning for Missouri
The National Association for the Promotion of Farms has issued a warning for people of color who travel to Missouri because of the state's discriminatory policies and racist attacks.
"The NAACP Travel Advisory for the State of Missouri, which is scheduled to take effect until August 28, 2017, calls on African-American travelers, visitors and Missourians to pay special attention and exercise extreme caution when traveling through the state, as a number of questionable, race-based incidents have recently occurred throughout the state and are noted here," the group's statement read.
A recent Missouri law that makes it harder for people to win discrimination lawsuits, as well as the state's law enforcement agencies that disproportionately target minorities, prompted the group to issue the travel warning, the NAACP said.
"They have civil rights violations that happen to people.
They are stopped because of their skin color, they are beaten up or killed, "the president of the Missouri NAACP, Rod Chapel, told the Kansas City Star.
'We're hearing complaints at such frequency that we haven't heard before. '
It is the first such warning issued by the Organization for a State in the United States.
The group cited incidents such as racist offences against black students at the University of Missouri and the death of Tory Sanders, 28, a black man from Tennessee.
Sanders died in questionable circumstances earlier this year after running out of gasoline while traveling through the state and being taken into custody by Missouri police without being charged with a crime.
The council also points to a recent report by the Missouri Attorney General's Office showing that black motorists in the state are 75 percent more likely to be stopped than whites.
"The recommendation is that people be aware and inform their families, friends and co-workers about what might happen in Missouri," chapel said.
"People have to be ready, whether they bring deposit money or let relatives know that they are going through the state."
Missouri recorded 100 hate crimes in 2015, according to the latest figures from the FBI's hate crime reporting program, which ranks the state 16th in the country in terms of the number of such violations.
The travel warning is also a response to a new Missouri law that would make it harder to sue a company for housing or employment discrimination.
Earlier, the American Civil Liberties Union (ACLU) issued travel warnings for Texas and Arizona after states passed immigration laws requiring local law enforcement to arrest people for immigration violations, which the ACLU says will help further racial profiling.
Travel alerts are usually issued by the State Department for other countries, but lately interest groups have used the measure in response to certain laws and trends within the U.S.
Electric fence project is basic
The Fischerbacher Weidezaun-Project is a success and will be continued in the coming year.
The CDU members Peter Weiß and Kordula Kovac got a picture on the spot.
At the Richtgrabenhof of the Schwendemann family, representatives of the Fischerbacher CDU local association met with Mayor Thomas Schneider and the two deputies.
Markus Schwendemann initially provided insights into his farm, where he farms 55 hectares of meadows and 40 hectares of forest, supplies 40 dairy cows and just as many young animals, and hosts holiday guests in five apartments.
For him, "Schwarzwaldmilch" is the alternative marketing route for the milk of his cows, says Schwendemann.
Only regional dishes are on the table of holiday guests.
Since he is also BLHV chairman of Fischerbach, he described the electric fence project as "fundamentally important."
I cannot make a profit on such steep areas.
The grassland area provides a lot of work, a lot of work has to be done with the engine sense.
"We are struggling here with the conditions and with the topography," Schwendemann stressed.
In 2003, he took over the parental business and managed it without chemical plant protection and fertilizer, even if he did not run a biofarm.
For Mayor Thomas Schneider, it was clear: "the municipality has recognised the importance of the electric fence project and is pleased with the Leader funding."
Farmers would have to be supported at a cost of between EUR 3 000 and EUR 4 000 per hectare.
It is hoped that the afforestation of the land could be dragged out for quite some time.
"The greatest treasure we have is the people who work in agriculture and are thinking about how to continue their business," said the mayor.
Farmers are often criticised for their support
In 2013, a total of 25 plots of eleven participants were fenced off via the fence promotion.
The open area thus amounts to 40 hectares, which may not be reforested until at least 2023.
Of the total cost of 71 500 euros, half was supported by Leader, and a quarter was contributed by the municipality of Fischerbach.
As a result, the farmers were left with 12 610 euros, which they had to invest in 16.2 kilometres of grazing fence.
"The big advantage is the longevity of the fences," explained Schwendemann.
Kordula Kovac stressed: "you are doing a really good job, without farmers and rural women it is not possible."
For Peter Weiß, too, it was clear: "this is where the best food is produced."
The Land Management Directive (LPR) Baden-Wuerttemberg makes it possible for the state to support fences.
The aim is to keep the Black Forest open.
A prerequisite is the existence of a funding framework defined by the LPR.
In addition, the area to be fenced must not be traceable (slope gradient above 35 percent) and must therefore be threatened by afforestation.
The proposal for the Fischerbacher Weidezaunprojekt 2018 is currently being submitted by eight participants with 24 plots, who want to build almost 16 kilometres of fence.
The disclosure area corresponds to 35 hectares, the calculated costs are more than 112 000 euros.
Is nothing sacred anymore?
Now the Trump White House is taking down the Statue of Liberty
The surest sign of regime change is when they begin to attack the statues.
Americans value them as much as anyone else - hence the carefully orchestrated overthrow of Saddam Hussein in Baghdad's Firdos Square in 2003.

100
src/ref.txt Normal file
View file

@ -0,0 +1,100 @@
Munich 1856: Four maps that will change your view of the city
A mental asylum, where today young people are said to meet.
A crypt chapel, where they are now digging tunnels for the S-Bahn.
Allotment holders cultivate the soil of former farmers.
The oldest official map of Munich brings captivating stories to light.
It is annoying when geographical maps are not up-to-date.
Anyone who has ever got worked up because the car's sat-nav is showing a green field instead of a bypass knows that.
The historical maps of the digital BayernAtlas, an offering from the State Government's Geoportal Bayern, are anything but up-to-date – and yet it is precisely for this reason that they are so informative.
Especially when one compares them with current online maps.
Then it becomes clear how the towns and municipalities in the distribution area of Munich's Merkur newspaper have changed since the 19th century.
The digital BayernAtlas is a patchwork quilt that one doesn't quite recognise for what it is at first.
Within the large overview map, were worked in detailed maps for towns and municipalities, so-called original-lay-of-the-land sheets.
According to information from the Bavarian Surveying and Mapping Authority, these came into being between 1810 and 1864 and arise out of the first survey of Bavaria.
Their commissioner: Napoleon.
The French army commander had occupied Bavaria since the spring in 1800, during the War of the Second Coalition with Austria.
Napoleon wanted to gain an overview of the occupied land and commissioned a topographical survey – naturally for military purposes.
Even if the French troops finally retreated with the Treaty of Lunéville from 9th February 1801: it was the current neighbours who had the idea to create a comprehensive map of Bavaria.
Elector Max IV. Joseph propelled the project further forward, founding the "Topographical Bureau" – and with that, the Bavarian Surveying and Mapping Authority.
The original lay-of-the-land sheet for Munich originates from 1856 and, like all the others, can be found on the website of the Bavarian State Library.
The comparison of past/present is most impressive taking the example of the state capital.
The city has stretched out in all directions and coalesced.
An example of this is Untersendling, which still appears relatively isolated on the map from 1856.
Sendling was incorporated as an urban district of Munich 21 years after the map was created.
Today, Untersendling lies in the middle of the city.
Around 1856, Untersendling's houses are concentrated along what is today Plinganserstraße, which was first named in 1878 after Georg Sebastian Plinganser, a ringleader of the Bavarian farmers' revolt against Austria in 1705.
Where the West Park is today, one sees nothing but field – at first sight: upon zooming in, a more detailed small-scale map with dozens of lines and numbers appears: hand-numbered plots from the so-called "original record", from which the property tax was derived.
Today, it is no longer farmers who occupy small parcels there – rather, it is allotment holders of the "Land in Sonne" facility.
A large green area is apparent to the north of Untersendling.
Theresienwiese, since 1810 the scene of Oktoberfest, was in the 19th century still actually a "Wiese" – a meadow – and not a concrete wasteland.
On the other side of the Isar, the historical map gives an insight into the history of psychiatric medicine.
In the original lay-of-the-land sheet, in the area where today the borough of Au-Haidhausen is, one can make out not only the street name "am Gasteig" or the elevation "Auf der Lüften" – but can actually read in large letters "mental asylum".
Yes, here on Auerfeldstraße, they once imprisoned people with delusions.
The Munich district mental asylum "Auf der Auer Lüften" had come to life there in 1858/59.
From 1872, Dr. Bernhard von Gudden led the institution – the psychiatrist who is associated with the death of King Ludwig II.
"He was progressive for his time and ended, for example, the practice of chaining up the so-called mad men.
What is more, he forbade the staff from using violence against the inmates," says Hermann Wilhelm, leader of the Haidhausen Museum, which has been busying itself for an exhibition with the establishment.
Originally there was place for 280 patients; later this was expanded to 500 places.
At some point even this capacity was no longer enough – the reason why the district mental asylum in Haar was constructed in 1905.
The Isar Amper Clinic, East Munich, has been located there since 2006, one of the largest psychiatric units in Germany.
On the plot of the former Auer institution, young people now meet in the Munich Salesianum.
The order acquired the building complex in 1920 and converted it into a youth centre with training programmes.
To the west lies the football pitch of TSV München Ost, and to the east, residential houses from the 1980s.
In 2015, the "Casa Don Bosco" nursery was built – in the architectural style of the original mental asylum building, which had been destroyed in the Second World War in bomb attacks.
Marienplatz was recorded as the "main square" in around 1856, and Kaufingerstraße was called "Kaufinger Gasse".
The Second World War also brought a decisive change to the heart of Munich.
The bombs of the Allies tore many holes in the city centre in December 1944.
One big one remains up to this day – at the Marienhof.
The English Institute, small commercial premises and a crypt chapel from the Middle Ages once stood in the Marienhof, where these days, preparations for the second S-Bahn main line are underway, with first digging to follow on 5th April 2017.
You can use the slider control to compare the old Munich map with the aerial view from today – and to fill in the hole at the Marienhof.
Civil rights group issues travel warning for Missouri
The National Association for the Advancement of Colored People has put out an alert for people of color traveling to Missouri because of the state's discriminatory policies and racist attacks.
"The NAACP Travel Advisory for the state of Missouri, effective through August 28th, 2017, calls for African American travelers, visitors and Missourians to pay special attention and exercise extreme caution when traveling throughout the state given the series of questionable, race-based incidents occurring statewide recently, and noted therein," the group's statement reads.
A recent Missouri law making it harder for people to win discrimination lawsuits, as well as the state's law enforcement disproportionately targeting minorities prompted the group to issue the travel alert, the NAACP said.
"You have violations of civil rights that are happening to people.
They're being pulled over because of their skin color, they're being beaten up or killed," the president of the Missouri NAACP, Rod Chapel, told The Kansas City Star.
"We are hearing complaints at a rate we haven't heard before."
It is the first such warning that the organization has issued for a state in the US.
The group cited incidents such as racial slurs against black students at the University of Missouri and the death of Tory Sanders, 28, a black man from Tennessee.
Sanders died under questionable circumstances earlier this year after he ran out of gas while traveling through the state, and was taken into custody by Missouri police without being accused of a crime.
The advisory also points to a recent report by the Missouri Attorney General's Office showing that black drivers in the state were 75 percent more likely to be pulled over than whites.
"The advisory is for people to be aware, and warn their families and friends and co-workers of what could happen in Missouri," Chapel said.
"People need to be ready, whether it's bringing bail money with them, or letting relatives know they are traveling through the state."
Missouri recorded 100 hate crimes in 2015, according to the latest figures from the FBI's hate crime reporting program, ranking the state at 16th in the country in terms of the number of such violations.
The travel warning is also a response to a new Missouri law that would make it more difficult to sue a business for housing or employment discrimination.
Previously, the American Civil Liberties Union (ACLU) had issued travel advisories for Texas and Arizona after the states passed immigration enforcement laws requiring local law enforcement to detain people on immigration violations which the ACLU said would increase racial profiling.
Travel warnings are usually issued by the State Department for other countries, but lately, advocacy groups have resorted to the measure in response to certain laws and trends inside the US.
Pasture fence project is fundamental
The Fischerbach pasture fence project is a successful project and will be continued next year.
Members of the Bundestag for the CDU Peter Weiß and Kordula Kovac got a first-hand view.
Representatives from the Fischerbach local CDU group met with mayor Thomas Schneider and the two representatives at Rechtgrabenhof farm, belonging to the Schwendemann family.
To start with, Markus Schwendemann gave insights into his business, in which he cultivates 55 hectares of pasture and 40 hectares of woodland, keeps 40 milk cows and just as many young stock, and accommodates five apartments of holiday makers.
Schwendemann says that, for him, there is no alternative to the "Black Forest milk" marketing channel he uses for the milk from his cows.
Only regional produce is served to the holiday-makers.
Since he is also BLHV chairman for Fischerbach, he described the pasture fence project as "fundamentally important".
I can't make a profit on that kind of steep-sloping ground.
Grassland areas make for a lot of work, a lot has to be done with the string trimmer.
"We have to grapple with the conditions and with the topography here," stressed Schwendemann.
In 2003, he took over his parents' business and he manages this without chemical crop protection and fertilisers, even if it's not an organic farm that he's running.
For Mayor Thomas Schneider, it was clear that "the community has recognised the importance of the pasture fence project and is happy to have the Leader funding".
The farmers need support to the tune of 3000 to 4000 euros per hectare.
One hopes that with that, afforestation of the area could then be drawn out for a long while.
"The biggest asset that we have are the people who work in agriculture and who give thought to how to bring their businesses forward," stressed the mayor.
The farmers are often the target of criticism thanks to the funding.
Using the pasture fence funding, a total of 25 plots from eleven participants were fenced off in 2013.
The areas kept open with these amount to 40 hectares, which may not be afforested until at least 2023.
Of the total costs, to the tune of 71,500 euros, half was funded via Leader, and a quarter was borne by the community of Fischerbach.
The farmers were thus left with 12,610 euros, which they had to invest in the 16.2-kilometre pasture fence.
"The great advantage is the longevity of the fences," explained Schwendemann.
Kordula Kovac stressed: "You're doing a really good job. Without farmers we'd not be able to function."
Peter Weiß is also standing firm: "First-class foodstuffs are produced here."
The Baden-Württemberg directive on countryside conservation (LPR) makes the government funding of pasture fences possible.
Its goal is to keep the Black Forest open.
A prerequisite is the existence of one of the funding scenarios as outlined by the LPR.
In addition, the fenced-off area is not allowed to be farmable with a tractor (gradient of over 35 per cent) and must therefore be threatened with afforestation.
For the 2018 Fischerbach pasture fence project, an application is currently underway from eight participants with 24 plots who would like to build just shy of 16 kilometres of fence.
The areas of land to be kept open equate to 35 hectares; the calculated costs amount to more than 112,000 euros.
Is nothing sacred?
Now Trump's White House is targeting the Statue of Liberty
The surest mark of regime change is when they start attacking the statues.
Americans appreciate this as well as anyone - hence the carefully stage-manged toppling of Saddam Hussein in Firdos square in Baghdad in 2003.