Elamur Research⌕
☰
home / papers / 2018 / FinSentiA: Sentiment Analysis in English Financial Microblogs

FinSentiA: Sentiment Analysis in English Financial Microblogs

Paper · 2018 · Thomas Gaillat, Annanda Sousa, Manel Zarrouk et al.

Summary

The paper targets investor sentiment in English financial microblogs, a domain where most prior work assigned labels to whole sentences or documents, or scored only text spans that had already been isolated by hand. The authors built a classifier for the SSIX platform that works on complete, unedited messages. It first finds the stocks (cashtags) a message mentions and cuts the text into segments, so that each entity is paired with its own context. It then predicts a real-valued bearish-to-bullish score in [-1, 1] for each entity. Training and testing use the SemEval-2017 Task 5 Gold Standard of StockTwits/Twitter posts. Segments are represented with bag-of-words and lexicon-based features, and these feed a network that combines LSTM and dense layers. The best version, 2.2, gets an entity-level cosine similarity of about 0.63 and an F1 of about 0.85 when scores are collapsed into two polarity classes. This is well above a simple baseline. The work suggests that fine-grained, entity-level sentiment can be estimated from noisy social media text in a realistic setup.

Key points

  • Data: the SemEval-2017 Task 5 corpus contains 2,002 English microblog posts, and its public release keeps 1,336 StockTwits posts. Each cashtag carries one score on [-1, 1]. Annotator agreement was Fleiss's Kappa 0.69 and Krippendorff's alpha 0.61. The authors used an 80/20 train/test split, with 1,761 training messages.
  • Features: bag of words, SentiWordNet, AFINN (2,477 entries scored from -5 to 5) and the Vader score. Word2vec and several other lexicons were tried but did not end up in the best combination.
  • Architecture: the BoW and SentiWordNet vectors each go through their own stack of five LSTM layers. The two outputs are concatenated with the AFINN and Vader values, passed through seven dense layers with 25% dropout, and the network outputs one value in [-1, 1].
  • Results: the baseline (Word2vec plus SentiWordNet with a single hidden layer) reached cosine 0.3352, accuracy 0.7801 and F1 0.8098. Version 2.2 reached cosine 0.6269, accuracy 0.8028 and F1 0.8522. Under the message-level SemEval metric it scored 0.713, against 0.733 for Jiang et al.
  • Ablation: adding the entity-aware message splitting raised cosine from 0.57 to 0.62. The model rarely predicts extreme scores, because the training data contains very few of them.

Interesting ideas

  • I1. Cashtags that sit next to each other are grouped into one target, and targets separated by more than one word are handled separately, with the message cut between them (for example at "but"). This gives each entity its own stretch of text to be judged on.
  • I2. The system is evaluated on whole raw messages instead of gold-selected spans, and the preprocessing step is measured by running the model with and without it. This shows what each pipeline stage really contributes under realistic input.
  • I3. The authors report a continuous agreement measure alongside a binarized polarity accuracy and F1. The two-class view makes the results comparable with studies that use different label schemes, while the continuous view preserves the fine-grained signal.
  • I4. Plotting the predicted score distribution against the gold distribution showed that the model never reaches the extremes, and this traced back to sparse training data at those values.
  • I5. The authors note that cosine similarity can stay high even when signs disagree. This means a metric for sentiment must penalize polarity flips explicitly, not just measure vector closeness.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (frames): when attaching evaluative stance to entities, prag could cut at cc/conj boundaries in the UD tree instead of relying on word gaps, so each holder–target–polarity frame gets its own clause.
  • I2 (learn): any repair rule or preprocessing step could be scored by an on/off ablation over unfiltered text, which keeps progress claims honest.
  • I3 (answer): graded judgments such as degree of approval or certainty in fable QA could be scored both exactly and after collapsing to a coarse label, which separates near-misses from outright wrong-direction answers.
  • I4 (learn): comparing output and gold distributions could become a routine check that flags label regions where silver data is too thin to learn from.
  • I5 (learn): metrics for polarity-bearing outputs such as negation or speech-act force should count sign errors directly, perhaps through a negative control that flips polarity and confirms the score falls.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🔬Laboratoire d'Informatique, de Robotique et de Microélectronique de MontpellierMontpellier · France3 works · 1,590 citations🔬Insight SFI Research Centre for Data AnalyticsDublin · Ireland9 works · 543 citations🎓University College CorkCork · Ireland5 works · 24 citations🔬Linguistique Ingénierie et Didactique des LanguesRennes · France4 works · 14 citations🎓National University of Ireland, MaynoothMaynooth · Ireland5 works · 10 citations🎓Université Rennes 2Rennes · France3 works · 9 citations

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksDecision Sciences · 209 worksManagement Science and Operations Research · 108 worksEconomics, Econometrics and Finance · 17 worksFinance · 5 works

🧭 Topics 2Research subjects and languages the work is about.English · 2,263 worksSentiment & emotion · 972 works

Cite

Cite as Thomas Gaillat, Annanda Sousa, Manel Zarrouk and Brian Davis. 2018. FinSentiA: Sentiment Analysis in English Financial Microblogs. Actes de la Conférence TALN. Volume 1 - Articles longs, articles courts de TALN.

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{gaillat2018finsentia,
  title = {{FinSentiA: Sentiment Analysis in English Financial Microblogs}},
  author = {Thomas Gaillat, Annanda Sousa, Manel Zarrouk and Brian Davis},
  booktitle = {Actes de la Conférence TALN. Volume 1 - Articles longs, articles courts de TALN},
  year = {2018},
  doi = {10.13025/s8xp7d},
  url = {https://doi.org/10.13025/s8xp7d}
}

References

26 references, 1 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. A NTWEILER , W., & F RANK , M. Z. (2004). Is All That Talk Just Noise? The Information Content of Internet Stock Message Boards. The Journal of Finance, 59(3), 1259–1294. find
  2. B ACCIANELLA , S., E SULI , A., & S EBASTIANI , F. (2010). SentiWordNet 3.0: An Enhanced Lexical Resource for Sentiment Analysis and Opinion Mining. Presented at the Proceedings of the Seventh Conference on International Language Resources and Evaluation (LREC’10), Valletta, Malta: European Language Resources Association (ELRA).
  3. B LITZER , D. M. (2017). S&P 500® S&P Dow Jones Indices Index Methodology. New York, USA: S&P Dow Jones Indices. find
  4. B OLLEN , J., M AO , H., & Z ENG , X.J. (2011). Twitter mood predicts the stock market. Journal of Computational Science, 2(1), 1–8. find
  5. C ORTIS , K., F REITAS , A., D AUDERT , T., H UERLIMANN , M., Z ARROUK , M., H ANDSCHUH , S., & D AVIS , B. (2017). SemEval2017 Task 5: FineGrained Sentiment Analysis on Financial Microblogs and News. In Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval2017) (pp. 519–535). Vancouver, Canada: ACL. find
  6. D AVIS , B., C ORTIS , K., V ASILIU , L., K OUMPIS , A., M CDERMOTT , R., & H ANDSCHUH , S. (2016). Social Sentiment Indices Powered by XScores. In ALLDATA 2016 , The Second International Conference on Big Data, Small Data, Linked Data and Open Data. Lisbon, Portugal: International Academy, Research, and Industry Association (IARIA). find
  7. G AILLAT , T., Z ARROUK , M., F REITAS , A., & D AVIS , B. (2018). The SSIX Corpus: A Trilingual Gold Standard Corpus for Sentiment Analysis in Financial Microblogs. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018). Miyazaki, Japan: ELRA. find
  8. G HOSH , A., L I , G., V EALE , T., R OSSO , P., S HUTOVA , E., R EYES , A., & B ARNDEN , J. (2015). SemEval2015 Task 11: Sentiment Analysis of Figurative Language in Twitter. In Proceedings of the 9th International Workshop on Semantic Evaluation. Denver, USA: Association for Computational Linguistics. find
  9. G HOSHAL , D., B HATNAGAR , S., E KBAL , A., S HAD A KHTAR , M., & B HATTACHARYYA , P. (2017). IITP at SemEval2017 Task 5: An Ensemble of Deep Learning and Feature Based Models for Financial Sentiment Analysis. In Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval2017). Vancouver, Canada: Association for Computational Linguistics. find
  10. H UTTO , C. J., & G ILBERT , E. (2014). VADER: A Parsimonious RuleBased Model for Sentiment Analysis of Social Media Text. In Eighth International AAAI Conference on Weblogs and Social Media. Ann Arbor, Michigan, USA: University of Michigan. find
  11. J IANG , M., L AN , M., & W U , Y. (2017). ECNU at SemEval2017 Task 5: An Ensemble of Regression Algorithms with Effective Features for FineGrained Sentiment Analysis in Financial Domain. In Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval 2017) (pp. 888–893). Vancouver, Canada: Association for Computational Linguistics. find
  12. J URAFSKY , D., & M ARTIN , J. H. (2009). Speech and Language Processing (2nd Edition). Upper Saddle River, NJ, USA: PrenticeHall, Inc. find
  13. L IU , B. (2012). Sentiment Analysis and Opinion Mining. San Rafael, Calif.: Morgan & Claypool Publishers. find
  14. L OUGHRAN , T., & M C D ONALD , B. (2011). When is a Liability not a Liability? Textual Analysis, Dictionaries, and 10Ks. The Journal of Finance, 1(66), 35–66. find
  15. M ALO , P., S INHA , A., T AKALA , P., A HLGREN , O., & L APPALAINEN , I. (2013). Learning the Roles of Directional Expressions and Domain Concepts in Financial News Analysis. In Proceedings of the 2013 IEEE 13th International Conference on Data Mining Workshops (pp. 945–954). Washington, DC, USA: IEEE Computer Society. find
  16. M ALO , P., S INHA , A., T AKALA , P., K ORHONEN , P., & W ALLENIUS , J. (2013). Good Debt or Bad Debt: Detecting Semantic Orientations in Economic Texts. Journal of the Association for Information Science and Technology, 65(4), 782–796. find
  17. M OHTARAMI , M ITRA , H ADI A MIRI , M AN L AN , T HANH P HU T RAN , AND C HEW L IM T AN . (2012). Sense Sentiment Similarity: An Analysis. In Proceedings of the TwentySixth AAAI Conference on Artificial Intelligence, 1706–1712. AAAI’12. Toronto, Ontario, Canada: AAAI Press. find
  18. M ORENO O RTIZ , A., & F ERNÁNDEZ C RUZ , J. (2015). Identifying Polarity in Financial Texts for Sentiment Analysis: A Corpusbased Approach. Procedia Social and Behavioral Sciences, 198, 330–338. find
  19. N IELSEN , F. Å. (2011). A new ANEW: Evaluation of a word list for sentiment analysis in microblogs. In Proceedings of the ESWC2011 Workshop on ‘Making Sense of Microposts’: Big things come in small packages (Vol. 718, pp. 93–98). Heraklion, Greece: CEUR Workshop Proceedings. find
  20. O’H ARE , N., D AVY , M., B ERMINGHAM , A., F ERGUSON , P., S HERIDAN , P., G URRIN , C., & S MEATON , A. F. (2009). Topicdependent Sentiment Analysis of Financial Blogs. In Proceedings of the 1st International CIKM Workshop on Topicsentiment Analysis for Mass Opinion (pp. 9–16). New York, USA: ACM. find
  21. P ANG , B., & L EE , L. (2008). Opinion Mining and Sentiment Analysis. Found. Trends Inf. Retr., 2(1–2), 1–135. find
  22. S CHUMAKER , R. P., Z HANG , Y., H UANG , C.N., & C HEN , H. (2012). Evaluating sentiment in financial news articles. Decision Support Systems, 53(3), 458–464. find
  23. S PRENGER , T. O., T UMASJAN , A., S ANDNER , P. G., & W ELPE , I. M. (2014). Tweets and Trades: the Information Content of Stock Microblogs. European Financial Management, 20(5), 926–957. find
  24. S RIVASTAVA , N., H INTON , G., K RIZHEVSKY , A., S UTSKEVER , I., & S ALAKHUTDINOV , R. (2014). Dropout: A Simple Way to Prevent Neural Networks from Overfitting. Journal of Machine Learning Research, 15, 1929–1958. find
  25. T ETLOCK , P. C. (2007). Giving Content to Investor Sentiment: The Role of Media in the Stock Market. The Journal of Finance, 62(3), 1139–1168. find
  26. W IEBE , J., W ILSON , T., & C ARDIE , C. (2005). Annotating Expressions of Opinions and Emotions in Language. Language Resources and Evaluation, 39(2–3), 165–210. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/gaillat-2018-finsentia-sentiment-analysis-english-financial/