Commit 43d7addb by ana

adding datafolder, scripts for scoring textfiles per sentece/word

parent ea143895
This diff could not be displayed because it is too large.
Part I
Chapter I
It was a bright cold day in April, and the clocks were striking thirteen. Winston Smith, his chin nuzzled into his breast in an effort to escape the vile wind, slipped quickly through the glass doors of Victory Mansions, though not quickly enough to prevent a swirl of gritty dust from entering along with him. The hallway smelt of boiled cabbage and old rag mats. At one end of it a coloured poster, too large for indoor display, had been tacked to the wall. It depicted simply an enormous face, more than a metre wide: the face of a man of about forty-five, with a heavy black moustache and ruggedly handsome features. Winston made for the stairs. It was no use trying the lift. Even at the best of times it was seldom working, and at present the electric current was cut off during daylight hours. It was part of the economy drive in preparation for Hate Week. The flat was seven flights up, and Winston, who was thirty-nine and had a varicose ulcer above his right ankle, went slowly, resting several times on the way. On each landing, opposite the lift shaft, the poster with the enormous face gazed from the wall. It was one of those pictures which are so contrived that the eyes follow you about when you move. BIG BROTHER IS WATCHING YOU, the caption beneath it ran.
Inside the flat a fruity voice was reading out a list of figures which had something to do with the production of pig-iron. The voice came from an oblong metal plaque like a dulled mirror which formed part of the surface of the right-hand wall. Winston turned a switch and the voice sank somewhat, though the words were still distinguishable. The instrument (the telescreen, it was called) could be dimmed, but there was no way of shutting it off completely. He moved over to the window: a smallish, frail figure, the meagreness of his body merely emphasized by the blue overalls which were the uniform of the Party. His hair was very fair, his face naturally sanguine, his skin roughened by coarse soap and blunt razor blades and the cold of the winter that had just ended.
Outside, even through the shut window-pane, the world looked cold. Down in the street little eddies of wind were whirling dust and torn paper into spirals, and though the sun was shining and the sky a harsh blue, there seemed to be no colour in anything, except the posters that were plastered everywhere. The black-moustachio’d face gazed down from every commanding corner. There was one on the house-front immediately opposite. BIG BROTHER IS WATCHING YOU, the caption said, while the dark eyes looked deep into Winston’s own. Down at street level another poster, torn at one corner, flapped fitfully in the wind, alternately covering and uncovering the single word INGSOC. In the far distance a helicopter skimmed down between the roofs, hovered for an instant like a bluebottle, and darted away again with a curving flight. It was the police patrol, snooping into people's windows. The patrols did not matter, however. Only the Thought Police mattered. Behind Winston's back the voice from the telescreen was still babbling away about pig-iron and the overfulfilment of the Ninth Three-Year Plan. The telescreen received and transmitted simultaneously. Any sound that Winston made, above the level of a very low whisper, would be picked up by it; moreover, so long as he remained within the field of vision which the metal plaque commanded, he could be seen as well as heard. There was of course no way of knowing whether you were being watched at any given moment. How often, or on what system, the Thought Police plugged in on any individual wire was guesswork. It was even conceivable that they watched everybody all the time. But at any rate they could plug in your wire whenever they wanted to. You had to live - did live, from habit that became instinct - in the assumption that every sound you made was overheard, and, except in darkness, every movement scrutinised.
Winston kept his back turned to the telescreen. It was safer; though, as he well knew, even a back can be revealing. A kilometre away the Ministry of Truth, his place of work, towered vast and white above the grimy landscape. This, he thought with a sort of vague distaste - this was London, chief city of Airstrip One, itself the third most populous of the provinces of Oceania. He tried to squeeze out some childhood memory that should tell him whether London had always been quite like this. Were there always these vistas of rotting nineteenth-century houses, their sides shored up with baulks of timber, their windows patched with cardboard and their roofs with corrugated iron, their crazy garden walls sagging in all directions? And the bombed sites where the plaster dust swirled in the air and the willowherb straggled over the heaps of rubble; and the places where the bombs had cleared a larger patch and there had sprung up sordid colonies of wooden dwellings like chicken-houses? But it was no use, he could not remember: nothing remained of his childhood except a series of bright-lit tableaux, occurring against no background and mostly unintelligible.
-1.004391
I was guiltless, but I had indeed drawn down a horrible curse upon my head, as mortal as that of crime.
-1.006065
We sat late.
-1.007096
said the old man.
-1.028012
The agonies of remorse poison the luxury there is otherwise sometimes found in indulging the excess of grief.
-1.036734
Blasted as thou wert, my agony was still superior to thine, for the bitter sting of remorse will not cease to rankle in my wounds until death shall close them forever.
-1.045306
Man, you shall repent of the injuries you inflict.
-1.050177
In one corner, near a small fire, sat an old man, leaning his head on his hands in a disconsolate attitude.
-1.055839
I saw him on the point of repeating his blow, when, overcome by pain and anguish, I quitted the cottage, and in the general tumult escaped unperceived to my hovel.
-1.060125
Her mild eyes seemed incapable of any severity or guile, and yet she has committed a murder.
-1.063721
Never did I behold a vision so horrible as his face, of such loathsome yet appalling hideousness.
-1.085048
But since the murderer has been discovered--The murderer discovered!
-1.085386
I may die, but first you, my tyrant and tormentor, shall curse the sun that gazes on your misery.
-1.085582
More miserable than man ever was before, why did I not sink into forgetfulness and rest?
-1.090738
Yet I am certainly unjust.
-1.097396
This sound disturbed an old woman who was sleeping in a chair beside me.
-1.099525
Chapter 12 I lay on my straw, but I could not sleep.
-1.109658
Saying this, he suddenly quitted me, fearful, perhaps, of any change in my sentiments.
-1.136478
The poor victim, who on the morrow was to pass the awful boundary between life and death, felt not, as I did, such deep and bitter agony.
-1.1452
Alas!
-1.1452
Alas!
-1.1452
Alas!
-1.147289
I paused.
-1.148148
Was there no injustice in this?
-1.148477
Yet mine shall not be the submission of abject slavery.
-1.154061
Why did you form a monster so hideous that even YOU turned from me in disgust?
-1.154612
A sister or a brother can never, unless indeed such symptoms have been shown early, suspect the other of fraud or false dealing, when another friend, however strongly he may be attached, may, in spite of himself, be contemplated with suspicion.
-1.155108
My swelling heart involuntarily pours itself out thus.
-1.164905
I am the assassin of those most innocent victims; they died by my machinations.
-1.172458
I shall no longer see the sun or stars or feel the winds play on my cheeks.
-1.181726
In a thousand spots the traces of the winter avalanche may be perceived, where trees lie broken and strewed on the ground, some entirely destroyed, others bent, leaning upon the jutting rocks of the mountain or transversely upon other trees.
-1.198924
Or whither does your senseless curiosity lead you?
-1.20918
The poor that stopped at their door were never driven away.
-1.215336
Mans yesterday may neer be like his morrow; Nought may endure but mutability!
-1.230786
Devil, cease; and do not poison the air with these sounds of malice.
-1.240057
Elizabeth had caught the scarlet fever; her illness was severe, and she was in the greatest danger.
-1.240959
I writhed under his words, yet dared not exhibit the pain I felt.
-1.245818
I passed the night wretchedly.
-1.248401
His jaws opened, and he muttered some inarticulate sounds, while a grin wrinkled his cheeks.
-1.253795
Felix trembled violently as he said this.
-1.256616
I remained silent.
-1.263614
I was a poor, helpless, miserable wretch; I knew, and could distinguish, nothing; but feeling pain invade me on all sides, I sat down and wept.
-1.282275
He became the victim of its weakness.
-1.285592
Fear overcame me; I dared no advance, dreading a thousand nameless evils that made me tremble, although I was unable to define them.
-1.302078
Fiend that thou art!
-1.311546
Let the cursed and hellish monster drink deep of agony; let him feel the despair that now torments me.
-1.3167
The wet wood which I had placed near the heat dried and itself became inflamed.
-1.330364
A frightful selfishness hurried me on, while my heart was poisoned with remorse.
-1.335277
I was alone; none were near me to dissipate the gloom and relieve me from the sickening oppression of the most terrible reveries.
-1.343951
I sat down, and a silence ensued.
-1.36678
As my sickness quitted me, I was absorbed by a gloomy and black melancholy that nothing could dissipate.
-1.370631
I then paused, and a cold shivering came over me.
-1.385116
I do not fear to die, she said; that pang is past.
-1.388903
Tears, unrestrained, fell from my brothers eyes; a sense of mortal agony crept over my frame.
-1.402591
I shall die.
-1.404472
His limbs were nearly frozen, and his body dreadfully emaciated by fatigue and suffering.
-1.410327
Cursed, cursed be the fiend that brought misery on his grey hairs and doomed him to waste in wretchedness!
-1.412762
What do these sounds portend?
-1.41817
Geneva, March 18, 17--.
-1.425913
Again do I vow vengeance; again do I devote thee, miserable fiend, to torture and death.
-1.431747
Had my eyes deceived me?
-1.450231
They spurn and hate me.
-1.460187
Then, overcome by fatigue, I lay down among some straw and fell asleep.
-1.500029
And do not you fear the fierce vengeance of my arm wreaked on your miserable head?
-1.545846
Man!
-1.578741
Justine shook her head mournfully.
-1.579585
I paused; at length he spoke, in broken accents: Unhappy man!
-1.590892
How often did I imprecate curses on the cause of my being!
-1.611354
But now crime has degraded me beneath the meanest animal.
-1.622701
During the whole of this wretched mockery of justice I suffered living torture.
-1.647614
Wherefore not?
-1.648735
Shall I not then hate them who abhor me?
-1.654698
I never beheld anything so utterly destroyed.
-1.668905
My own agitation and anguish was extreme during the whole trial.
-1.674818
I remained motionless.
-1.683465
He threatened excommunication and hell fire in my last moments if I continued obdurate.
-1.686961
I, a miserable wretch, haunted by a curse that shut up every avenue to enjoyment.
-1.715548
The monster continued to utter wild and incoherent self-reproaches.
-1.731914
There he lies, white and cold in death.
-1.735126
Miserable himself that he may render no other wretched, he ought to die.
-1.775426
Oh, Frankenstein!
-1.804502
Why did I not die?
-1.822475
I could have torn him limb from limb, as the lion rends the antelope.
-1.823312
Why did I not then expire!
-1.860791
Leave me; I am inexorable.
-1.879978
I never saw a man in so wretched a condition.
-1.881685
I pitied Frankenstein; my pity amounted to horror; I abhorred myself.
-1.894665
I do refuse it, I replied; and no torture shall ever extort a consent from me.
-1.975587
Poor little fellow!
-1.97756
Or rather, stay, that I may trample you to dust!
-1.993792
I have endured incalculable fatigue, and cold, and hunger; do you dare destroy my hopes?Begone!
-10.273834
Begone!
-10.51603
Miserable, unhappy wretch!
-10.930867
Despair!
-12.273019
Ugly wretch!
-2.012111
Chapter 16 Cursed, cursed creator!
-2.070458
I did confess, but I confessed a lie.
-2.156648
Thus I might proclaim myself a madman, but not revoke the sentence passed upon my wretched victim.
-2.208422
Oh, not abhorred!
-2.255191
No guilt, no mischief, no malignity, no misery, can be found comparable to mine.
-2.27524
Poor, poor girl, is she the accused?
-2.345074
What a miserable night I passed!
-2.426911
I did not yet entirely know the fatal effects of this miserable deformity.
-2.487342
Geneva, May 12th, 17--.
-2.625245
Frankenstein
-2.679246
I never could survive so horrible a misfortune.
-2.722253
Nay, then I was not miserable.
-2.848968
No; I am not so selfish.
-2.911875
I am interrupted.
-3.021047
Poor William!
-3.069702
I trembled.
-3.091565
R.W.
-3.114359
Clerval!
-3.402281
Oh, no!
-3.683603
Suddenly a heavy storm of rain descended.
-3.789021
Poor girl!
-4.030508
You may hate, but beware!
-4.177622
I knocked.
-4.350249
I thought (foolish wretch!)
-4.381934
He struggled violently.
-4.595895
I am malicious because I am miserable.
-4.733167
Abhorred monster!
-4.976811
Poor Clerval!
-5.525419
Unfeeling, heartless creator!
-5.878955
No!
-6.044624
Do not despair.
-6.321552
I trembled violently, apprehending some dreadful misfortune.
-6.501103
Do not fear.
-7.414885
Wretch!
-8.231735
Scoffing devil!
-8.805615
Begone, vile insect!
-9.783166
Wretched devil!
-9.875328
Hypocritical fiend!
-9.878561
Hideous monster!
This diff could not be displayed because it is too large.
This diff could not be displayed because it is too large.
### Following this tutorial in Python3:
### https://gist.github.com/rspeer/ef750e7e407e04894cb3b78a82d66aed#file-how-to-make-a-racist-ai-without-really-trying-ipynb
'''
This script applies a trained model to textfiles.
It splits the text in sentences and predicts a sentiment score for each of the sentences / or for each of the words
The score & sentence/word are saved in a file, ordered from small to big scores
clean your text using adapting_the_reading_glasses.py
'''
import numpy as np
import pandas as pd
from sklearn.linear_model import SGDClassifier
# from sklearn.model_selection import train_test_split
# from sklearn.metrics import accuracy_score, classification_report, confusion_matrix, f1_score
from sklearn.externals import joblib
### FUNCTIONS
### LOAD GloVE word Embeddings
### ---------------------------
def load_embeddings(filename):
"""
Load a DataFrame from the generalized text format used by word2vec, GloVe,
fastText, and ConceptNet Numberbatch. The main point where they differ is
whether there is an initial line with the dimensions of the matrix.
"""
labels = []
rows = []
with open(filename, 'r') as infile:
for i, line in enumerate(infile):
# print(i)
#print(line)
items = line.rstrip().split(' ')
if len(items) == 2:
# This is a header row giving the shape of the matrix
continue
labels.append(items[0])
values = np.array([float(x) for x in items[1:]], 'f')
rows.append(values)
arr = np.vstack(rows)
return pd.DataFrame(arr, index=labels, dtype='f')
### ------------------------------------------
### ACTIONS
### LOAD EMBEDDINGS
embeddings = load_embeddings('data/glove.840B.300d.txt')
#embeddings = load_embeddings('data/glove.42B.300d.txt')
#embeddings = load_embeddings('data/glovesample.txt')
#filename = 'data/1984_all_stripped.txt'
#filename = 'data/1984_fragment.txt'
filename = 'data/frankenstein_for_machines.txt'
pos_output = filename.replace('.txt','_pos.txt')
neg_output = filename.replace('.txt','_neg.txt')
scored_words = filename.replace('.txt','_scored_words.txt')
### SCORE WORDS
### ---------------
# load model using joblib
loaded_model = joblib.load('sentiment_thermometer_glove.pkl')
# get text & turn into set of unique words
all_words = set()
with open(filename, "r") as source:
for line in source:
if line:
#print("line", line)
# return word list
words = line.strip().split()
# make set of unique words of the text
for word in words:
all_words.add(word)
# look for words of given sentence in word embeddings
vecs = embeddings.loc[all_words].dropna()
# find logarithmic scores
predictions = loaded_model.predict_log_proba(vecs)
# print("tot1", predictions[:, 1])
# print("tot0", predictions[:, 0])
log_odds = predictions[:, 1] - predictions[:, 0]
#print("log_odds", log_odds)
# print score for each word
word_scores = pd.DataFrame({'': log_odds}, index=vecs.index)
# write words+scores to file
word_scores.to_csv(scored_words, sep='\t', encoding='utf-8')
print("done")
\ No newline at end of file
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or sign in to comment