Friday, May 19, 2006

Tutorial: Bayesian Techniques for NLP

Title: Beyond EM: Bayesian Techniques for Human Language Technology Researchers
Date: 24 May 2006
Time: 9am - noon
Location: 4th floor conference room

Expectation Maximization (EM) has proved to be a great and useful technique for unsupervised learning problems in speech and language processing. Unfortunately, its range of applications is limited either by intractable E- or M-steps, or by its reliance on the maximum likelihood estimator. The natural language processing community typically resorts to ad-hoc approximation methods to get (some reduced form of) EM to apply to NLP tasks. However, many of the problems that plague EM can be solved with Bayesian methods, which are theoretically more well justified. In this tutorial, I discuss Bayesian methods as they can be used in natural language processing. The two primary foci of this tutorial are specifying prior distributions and performing the necessary computations to perform inference in Bayesian models. I focus on unsupervised techniques (for which EM is the obvious choice), but discuss supervised and discriminative techniques at the conclusion with pointers to relevant literature.

Depending on one's inference technique of choice, the math required to build Bayesian learning models can be difficult. Compounding this problem is the fact that current written tutorials on Bayesian techniques tend to focus on continuous-valued problems, a poor match for the high-dimension discrete world of text. This combination makes the cost of entrance to the Bayesian learning literature often too high. The goal of this tutorial is to provide sufficient motivation, intuition and vocabulary mapping so that one can easily understand recent papers in Bayesian learning that are published at conferences like NIPS, and increasingly at ACL. In addition to the standard tutorial materials (slides), this tutorial is accompanied by a technical report that spells out all the mathematic derivations in great detail, for those who wish to start research projects in this fields.

This tutorial should be accessible to anyone with a basic understanding of statistics. I use a query-focused summarization task as a motivating running example for the tutorial, which should be of interest to researchers in natural language processing and in information retrieval. Additionally, though the tutorial does not focus on speech problems, those attendees interested in graphical modeling techniques for automatic speech recognition might also find the tutorial of interest.

2 comments:

fish said...

When we are born our brains are like empty computers waiting to be fed information. As we grow our peers act as our programmers, they supply us with the knowledge which we channel through the conscious mind into the subconscious (our hard drive). The subconscious mind is the biggest hard drive ever developed - it stores everything we come in contact with and by no means is all of this information of a positive nature.
All that we have heard, touched, smelt, tasted and seen are stored in the recesses of our minds. The subconscious mind holds on to this information until we need to recall it. For example when you were young your curiosity lead you to investigate your surroundings. When you approached a substance that was dangerous, such as fire, your parents or guardians would most likely have rebuked or scolded you if you ventured too near the flame. Perhaps you may even recall an incident when you were physically burned. Your subconscious mind then began to relate scolding (or pain) with the intense heat of the fire and would therefore feed the feelings of the scolding incident back to you whenever you got too close to fire again, thus acting as an early warning system.
This is the mechanism used by our brains to learn. It is also the same method employed by the mind in every situation. The subconscious mind has a tendency to emulate what it sees - it tends to replicate its environment. This is why so many people find themselves in similar relationships and situations that they saw their parents in while they were growing up. Most people also hold very strongly or similar views of their parents.
Think of a time when you gave yourself praise. What words did you use? Do you use the same words that your parents or peers used when they were praising you? The same is applicable when you scold yourself.

Watch your internal dialogue. Look at it closely. It takes diligence to change the way you think. When you notice yourself thinking a negative chose to think the opposite. This way you neutralise the negative thought. Now the think the positive thought again! You have just reversed the negative thinking in that moment and remember you only have this moment. No other time exists!
Daydream about what might be. Imagine things they way you wish them to be. If you catch yourself thinking "this is just a daydream - a fantasy" then stop! Think the opposite. It is not a daydream it is your reality. Now think it again.

By doing this simple procedure you will begin to retrain your subconscious mind to think positively and you will ultimately begin to consciously create a life that dreams are made of! personal development plan

fish said...

Many people try to achieve goals. Most fail. Some strive, work hard and plan for all the details yet they achieve little or nothing at all. Others strive, work hard, plan and achieve huge success. Yet there are a few individuals who do little else than take small steps and seem to achieve a great deal with what seems like effortlessness. What is the difference between these people and which one would you like to be?
Most members of the human race fall into two categories - those who live in the past and those who live in the future. Most live in the past. Many of these are the people who achieve very little in their lives and are so fearful of the future that they dare not strike out to get anything. They are the under-achievers who hang onto bad episodes in their lives and either relive them time and again or look at new situations as similar potentialities. They say things like "all men are deceivers" or "all women are interested in is money" or "I can't do it. I tried before and it didn't work so why bother!". Due to bad experiences in the past they believe that all future events will turn out the same way if they dare to go after what they want.
The other type of person lives in the future. This type tends to create more of the things they want in life. They have a vision of where they want to go and exactly how they are going to get there. They work diligently at making concrete plans and they pursue those plans with a persistent ferocious appetite for success. These people are the high achievers - The Richard Branson and Bill Gates of the world. These people have much to teach us about setting and achieving goals.
However, there is a third type of person who almost goes unnoticed. They are the person who takes life in its stride and yet achieve most of what they want. I am sure you know of such a person in your life that just seems to saunter through life and yet they always come out on top. Or a person who you hear of that has decided to open a shop. You meet them a few months later and they have three shops all doing well! So what makes these people so successful and if they aren't living in the past and aren't living in the future where are they living?
I suppose you guessed it! Whether they are consciously aware of it or not they are living in the present. It is in the 'living' present that we have our greatest power. Everything happens in the present. You live your entire life there - even if your mind does not!

By becoming more aware of the present and by 'accepting' it as it is we are much more in control of our emotions and focus. When we live in the past we are fearful of making bad choices and/or getting hurt. We do not wish to recreate the past again! When we live in the future we can also be fearful of what might happen. But even if your future vision is full of power and worthy of working towards many people can, and often do, get stuck there. By constantly reaching for bigger and better goals they fail to enjoy what they have in the moment.
If you wish to start living a life that is almost effortless begin first by living in the present. Accept your situation the way it is and then you can enjoy what you have. Your focus changes from a memory of what was or a vision of what might be to a realization of what is. You become much more empowered to then see the beauty of life and also look at where you wish to make changes. But to make changes you must first accept the situation as it is. Trying to escape from your present only increases your focus on your problems by creating resistance to what is. Accept your life as it is now. Make no judgement, just accept it and then you will be free of doubt, worry, pain and fear. For you only experience these things when you live outside the 'moment'. subliminal