Sentiment analysis

Àíàëèç òîíàëüíîñòè òåêñòîâ

Âûëîìîâà Åêàòåðèíà Àëåêñååâíà: evylomova@gmail.com

Ìîñêâà, 2013

Âûëîìîâà Åêàòåðèíà Àëåêñååâíà: evylomova@gmail.com () Àíàëèç òîíàëüíîñòè òåêñòîâ Ìîñêâà, 2013 1 / 41

×àñòü 1. Experience project
Îïèñàíèå

Experience project: ïðîåêò, â ðàìêàõ êîòîðîãî ïîëüçîâàòåëè äåëÿòñÿ ñâîèìè èñòîðèÿìè.
Êàæäîé èñòîðèè ÷èòàòåëè ìîãóò âûñòàâèòü îäíó èç ïÿòè êàòåãîðèé è íàïèñàòü êîììåíòàðèé.

Ïðèìåð: I really hate being shy ... I just want to be able to talk to someone about anything and
everything and be myself ... That's all I've ever wanted.

Reactions: hugs: 1; rock: 1; teehee: 2; understand: 10; wow: 0;

Author age: 21

Author gender:female

Text group: friends


×àñòü 1. Äàííûå

Çàãðóçèì äàííûå:

e p = r e a d . c s v ( ' e p 3−c o n t e x t . c s v ' )

Çäåñü: Count - âñòðå÷àåìîñòü äàííîãî ñëîâà â ñîîòâåòñòâóþùåé Category, Group è ïðè óêàçàííûõ
Genger, Age
Total - îáùåå êîëè÷åñòâî ñëîâ ñîîòâåòñòâóþùåé Category, Group è ïðè óêàçàííûõ Genger, Age


×àñòü 1. Ðàáîòà ñ äàííûìè

Ìîæíî ïîëó÷èòü äàííûå ïî ëþáîìó èç ïàðàìåòðîâ:

l e v e l s ( ep$Word )


×àñòü 1. Ñëîâà è êàòåãîðèè

Ïîñìîòðèì, êàê ñîîòíîñÿòñÿ ñëîâà ñ êàòåãîðèÿìè:

f u n n y = e p C o l l a p s e d F r a m e ( ep , ' funny ' )
p l o t ( funny$Category , funny$Count , x l a b =' C a t e g o r y ' , y l a b =' Count ' , main ='
funny ' )



Ñòðàííî, ïðàâäà? À ÷òî íå òàê?


Íóæíà íîðìàëèçàöèÿ íà ðàçìåð êàòåãîðèè!

funny$Count / funny$Total
f u n n y = e p C o l l a p s e d F r a m e ( ep , ' funny ' , f r e q s=TRUE)
p l o t ( funny$Category , funny$Freq , x l a b =' C a t e g o r y ' , y l a b =' Count / T o t a l ' ,
main =' f u n n y ' )



Ãîðàçäî ëó÷øå!

×àñòü 1. Ê òåîðèè âåðîÿòíîñòåé

Freq ñîîòâåòñòâóåò óñëîâíîé âåðîÿòíîñòè P(word|category). Ýòî çíà÷åíèå âåñüìà ìàëî, ïîýòîìó
ïîñ÷èòàåì âåðîÿòíîñòü P(category|word) (âñå ïîìíÿò ôîðìóëó Áàéåñà?:-) ) .

funny$Freq / sum ( f u n n y $ F r e q )
f u n n y = e p C o l l a p s e d F r a m e ( ep , ' funny ' , f r e q s =TRUE , p r o b s=TRUE)
p l o t ( funny$Category , funny$Pr , x l a b =' C a t e g o r y ' , y l a b = ' ( C o u n t / T o t a l ) / sum (
Count / T o t a l ) ' , main =' f u n n y ' )

Âîïðîñ: Êàêèå åùå ñëîâà ìîãóò õîðîøî õàðàêòåðèçîâàòü êàêóþ-ëèáî èç êàòåãîðèé?


×àñòü 1. Ñðàâíèì çíà÷åíèÿ ñ îæèäàåìîé âåðîÿòíîñòüþ

f u n n y = e p C o l l a p s e d F r a m e ( ep , ' funny ' , f r e q s =TRUE , p r o b s=TRUE , o e=TRUE )

Êàê ïîñ÷èòàòü îæèäàåìîå çíà÷åíèå?

c a t e g o r y . p r o b s = ( f u n n y $ T o t a l / sum ( f u n n y $ T o t a l ) )
f u n n y . c o u n t = sum ( f u n n y $ C o u n t )
funny . expected = funny . count ∗ category . probs
funny . expected

Ïîñìîòðèì íà îòíîøåíèå:

( funny$observed / funny . expected ) − 1

Åñëè îíî ìåíüøå 0 - ñëîâî íåäîïðåäñòàâëåíî â êàòåãîðèè, áîëüøå - íàîáîðîò.


×àñòü 1. Ó÷åò êîíòåêñòà

p a r ( mfrow=c ( 1 , 3 ) )
e p P l o t ( ep , eptok , ' awesome ' , g e n d e r s =' male ' , p r o b s= )
T
e p P l o t ( ep , eptok , ' awesome ' , g e n d e r s =' f e m a l e ' , p r o b s= )
T
e p P l o t ( ep , eptok , ' awesome ' , g e n d e r s = ' unknown ' , p r o b s= )
T



p a r ( mfrow=c ( 2 , 3 ) )
for ( i in 1:5) { e p P l o t ( ep , eptok , ' awesome ' , a g e s=i , p r o b s= )
T }



Ïîñìîòðèì èçìåíåíèå ïàðàìåòðà äëÿ êàæäîé êàòåãîðèè îòäåëüíî:

e p C a t e g o r y B y F a c t o r P l o t ( ep , eptok , ' awesome ' , ' Gender ' , p r o b s=T , t y p e = 'b
')



Óïðàæíåíèå: ïîäáåðèòå ñëîâà äëÿ ëþáîãî ïàðàìåòðà (Age, Group, Gender), êîòîðûå
ïîä÷åðêèâàþò âàæíîñòü ïàðàìåòðà.



Ðàññêàçû ñî ñëîâîì drunk ñèëüíî çàâèñÿò îò âîçðàñòà àâòîðà

e p C a t e g o r y B y F a c t o r P l o t ( ep , eptok , ' drunk ' , ' Age ' , p r o b s=T , t y p e ='b ' )


×àñòü 1. Ïîñòðîåíèå ðåãðåññèîííîé ìîäåëè

Ïîïðîáóåì ïîñòðîèòü ýòó çàâèñèìîñòü ïðè ïîìîùè ëîãèñòè÷åñêîé ðåãðåññèè:

d r u n k = e p F u l l F r a m e ( ep , ' drunk ' , a g e=c ( 1 , 2 , 3 , 4 , 5 ) )
drunk$Age = as . numeric ( drunk$Age )
f i t . g l m = g l m ( c b i n d ( C o u n t , T o t a l −C o u n t ) ~ Category − 1 + Age , d a t a=d r u n k ,
f a m i l y =b i n o m i a l )
summary ( f i t . g l m )



Íàïèøåì ôóíêöèþ, ïðîãíîçèðóþùóþ çíà÷åíèÿ â çàâèñèìîñòè îò êàòåãîðèè è âîçðàñòà

FittedGlmFunc = function ( fit , category , age ) {
coefs = fit$coef
cat . coef = c o e f s [ [ paste ( ' Category ' , category , sep = ' ') ] ]
prediction = p l o g i s ( cat . coef + c o e f s [ [ ' Age ' ] ] ∗ age )
return ( prediction )
}

Âûçîâ ôóíêöèè:

F i t t e d G l m F u n c ( f i t . glm , ' wow ' , 1)



Âèçóàëèçèðóåì ïîëó÷åííûå çíà÷åíèÿ è ñðàâíèì èõ ñ èñòèííûìè:

p a r ( mfrow=c ( 2 , 3 ) )
cats = l e v e l s ( ep$Category )
for ( i in 1:5) {
e p P l o t ( ep , eptok , ' drunk ' , a g e= i )
for ( j in 1:5) {
val = F i t t e d G l m F u n c ( f i t . glm , cats [ j ] , i )
points ( j , val , c o l =' r e d ' , p c h =19)
}
}


×àñòü 1. Ïîäñ÷åò îæèäàåìîãî çíà÷åíèÿ


×àñòü 2

Àíàëèç ñëîâîñî÷åòàíèé "íàðå÷èå-ïðèëàãàòåëüíîå" íà ïðèìåðå ðåéòèíãîâûõ äàííûõ.


×àñòü 2. Äàííûå

Èñïîëüçóåì äàííûå èç ðàçëè÷íûõ ðåéòèíãîâûõ ñèñòåì(Amazon.com, OpenTable.com,
Goodreads.com, IMDB.com). Çàãðóçèì èõ:

d = read . csv ( ' ratings −a d v a d j . c s v ')
head ( d )


×àñòü 2. Èçâëå÷åíèå ïîäâûáîðîê

horrid = ratingFullFrame (d , ' horrid ' , t y p e s=NULL , m o d i f i e r s =NULL ,
m o d i f i e r . t y p e s=NULL , r a t i n g m a x =0)
nrow ( h o r r i d )
head ( h o r r i d )

Ñ óêàçàíèåì ìîäèôèêàòîðà:

horrid = ratingFullFrame (d , ' horrid ' , m o d i f i e r s =' a b s o l u t e l y ' )
nrow ( h o r r i d )
head ( h o r r i d )


×àñòü 2. Îöåíêà òîíàëüíîñòè îòäåëüíûõ ïðèëàãàòåëüíûõ

horrid = ratingCollapsedFrame (d , ' horrid ' , f r e q s =TRUE , p r o b s=TRUE )
horrid


×àñòü 2. Ãðàôèê òîíàëüíîñòè

p a r ( mfrow=c ( 1 , 2 ) )
ratingPlot (d , ' horrid ' , p r o b s=FALSE )
ratingPlot (d , ' horrid ' , p r o b s=TRUE)

Âîïðîñ: ïðåäëîæèòå ïðèëàãàòåëüíûå, êîòîðûå ìàêñèìèçèðóþò ñåðåäèíó ãðàôèêà.



Ïîïðîáóåì ñïðîãíîçèðîâàòü êàòåãîðèþ, èñõîäÿ èç ïðèëàãàòåëüíîãî. Äëÿ ýòîãî ïîñ÷èòàåì ìàò.
îæèäàíèå.

sum ( h o r r i d $ C a t e g o r y ∗ horrid$Pr )

Òî æå ñàìîå äåëàåì ôóíêöèÿ ExpectedCategory:

ExpectedCategory ( h o r r i d )

Äîáàâëåíèå îæèäàåìîãî çíà÷åíèÿ íà ãðàôèê:

ratingPlot (d , ' horrid ' , p r o b s=TRUE , e c=TRUE)


×àñòü 2. Ðåãðåññèîííàÿ ìîäåëü

Ïîïðîáóåì ïîñòðîèòü ìîäåëü äëÿ îöåíêè âåðîÿòíîñòè íàõîæäåíèÿ ñëîâà â êàòåãîðèè.

f i t . horrid = glm ( c b i n d ( h o r r i d $ C o u n t , horrid$Total −h o r r i d $ C o u n t ) ~
Category , f a m i l y =q u a s i b i n o m i a l , d a t a= h o r r i d )
f i t . horrid



Óëó÷øèì ìîäåëü, ñäåëàâ åå áîëåå êâàäðàòè÷íîé (äîáàâèì êâàäðàò êàòåãîðèè).

GlmWordQuadratic
function ( pf ) {
pf$Category2 = p f $ C a t e g o r y ^2
fit = g l m ( c b i n d ( C o u n t , T o t a l −C o u n t ) ~ Category + Category2 , f a m i l y=
quasibinomial , d a t a=p f )
return ( f i t )
}
p a r ( mfrow=c ( 2 , 2 ) )
ratingPlot (d , ' good ' , p r o b s=TRUE , m o d e l s=c ( G l m W o r d Q u a d r a t i c ) , ratingmax
=5 , y l i m=c ( 0 , 0.5) )
ratingPlot (d , ' good ' , p r o b s=TRUE , m o d e l s=c ( G l m W o r d Q u a d r a t i c ) , ratingmax
=10 , y l i m=c ( 0 , 0.3) )
ratingPlot (d , ' disappointing ' , p r o b s=TRUE , m o d e l s=c ( G l m W o r d Q u a d r a t i c ) ,
r a t i n g m a x =5 , y l i m=c ( 0 , 0.5) )
ratingPlot (d , ' disappointing ' , p r o b s=TRUE , m o d e l s=c ( G l m W o r d Q u a d r a t i c ) ,
r a t i n g m a x =10 , y l i m=c ( 0 , 0.3) )

×àñòü 3

Âåêòîðíûå ìîäåëè.


×àñòü 3. Èñõîäíûå äàííûå

Èñïîëüçóåòñÿ áàçà IMDB.
Íà÷àëüíûå äàííûå - ìàòðèöà òåðì x òåðì, ãäå ýëåìåíò ìàòðèöû - ýòî ÷àñòîòà ñîâñòðå÷àåìîñòè
äâóõ òåðìîâ â îäíîì êîíòåêñòå (äîêóìåíòå, ïðåäëîæåíèè è ò.ä.)

s o u r c e ( ' vsm . R ' )
i m d b = C s v 2 M a t r i x ( ' imdb −w o r d w o r d . c s v ' )
imdb [ 1 0 0 : 1 1 0 , 100:110]


×àñòü 3. Áàçîâûå ïîíÿòèÿ î âåêòîðàõ

Åâêëèäîâî ðàññòîÿíèå:
n

EuclideanDist (x , y ) = (x − y )2
i i

i =1

Äëèíà âåêòîðà:
n

VectorLength(x ) = (x )2i

i =1

Íîðìàëèçàöèÿ âåêòîðà - äåëåíèå êàæäîé êîìïîíåíòû íà äëèíó.
Êîñèíóñ óãëà ìåæäó âåêòîðàìè:

(x ) ∗ =1 (y )
n n

CosineDist (x , y ) = 1 − i =1 i i i

VectorLength(x ) ∗ VectorLength(y )


×àñòü 3. Ñåìàíòè÷åñêè áëèçêèå ñëîâà

d f = N e i g h b o r s ( imdb , ' happy ' )
head ( d f )


×àñòü 3. Ñåìàíòè÷åñêè áëèçêèå ñëîâà

Ïðîáëåìà:

a = c (1000 , 2000 , 3000)
b = c (1 , 2, 3)
a / sum ( a )
> [1] 0.1666667 0.3333333 0.5000000
b / sum ( b )
> [1] 0.1666667 0.3333333 0.5000000
LengthNorm ( a )
> [1] 0.2672612 0.5345225 0.8017837
LengthNorm ( b )
> [1] 0.2672612 0.5345225 0.801783


×àñòü 3. PMI - Pointwise mutual information

Êàê ýòîãî èçáåæàòü? - PMI!
p (x , y )
PMI (x , y ) = log
p (x ) ∗ p (y )
Íîðìèðîâêà PMI:

p (i , j ) min (
m
p (k , j ), n
p (k , j ))
NewPMI (i , j ) = pmi (i , j ) ∗ ∗ k =1 k =1
p (i , j ) + 1 min ( m

k =1 p (k , j ),
n

k =1 p (k , j )) + 1

Çäåñü p(i,j)=M/sum(M), M - ìàòðèöà òåðìîâ


×àñòü 3. PMI - Pointwise mutual information

i m d b . p p c d = PMI ( imdb , p o s i t i v e =TRUE , d i s c o u n t i n g=TRUE)
d f = N e i g h b o r s ( imdb . ppcd , ' happy ' , b y r o w=TRUE , d i s t f u n c =C o s i n e D i s t a n c e )
head ( d f )


×àñòü 3. Semantic orientation method

Îïèøåì äâà ìíîæåñòâà ñëîâ S1 è S2

Âûáåðåì ìåðó áëèçîñòè

Äëÿ íåêîòîðîãî ñëîâà w, ïîñ÷èòàåì ñóììó ðàññòîÿíèé äî âåêòîðîâ ìíîæåñòâ S1 è S2

Îöåíêà òîíàëüíîñòè - ðàçíèöà ìåæäó ñóììàìè ðàññòîÿíèé

n e g = c ( ' bad ' , ' nasty ' , ' poor ' , ' negative ' , ' unfortunate ' , ' wrong ' , '
inferior ')
p o s = c ( ' go o d ' , ' nice ' , ' excellent ' , ' positive ' , ' fortunate ' , ' correct ' ,
' superior ')
S e m a n t i c O r i e n t a t i o n ( imdb . ppcd , word =' g r e a t ' , s e e d s 1=n e g , s e e d s 2=p o s ,
d i s t f u n c =C o s i n e D i s t a n c e )
> 0.8923544
S e m a n t i c O r i e n t a t i o n ( imdb . p p c i , word =' h o r r i d ' , s e e d s 1=n e g , s e e d s 2=p o s ,
d i s t f u n c =C o s i n e D i s t a n c e )
> −0.04741898


Sentiment analysis

Empfohlen

Empfohlen

Weitere ähnliche Inhalte

Kürzlich hochgeladen

Kürzlich hochgeladen (8)

Empfohlen

Empfohlen (20)

Sentiment analysis