interactive learning

machine learning deals with the task of learning a function from observations of examples which have been labeled or unlabeled. This recovered function can be used to make predictions on the future new coming data.

However, most of previous ML algorithms do not interact with the environment, or other helpful resources that may improve the learning ability. A new topic appears recently, which is called interactive learning. Its idea has close connection with active learning and self-taught learning. The computer agent not only analyzes the data by utilizing its powerful computation ability, but also develops a kind of intelligent ability to actively seek new resources and interact with other objects in the world to improve learning ability. Human has this ability. When a baby is learning to speak, the first step he/she is trying to mimic the sound from his/her parents. At the same time, he/she can feel the feedback from the parents, such as appraise or disappointment. Based on such feedback, a baby will adjust his/her speaking. After the baby grows more mature, he/she is able to infer the intent of parents and proactively takes some action to attract parents or probe the feedback of parents.

Computer of course has larger advantage in computation capacity than human being, while it needs more advanced algorithms to become as intelligent as human being. That is an important purpose of artificial intelligence. The new research on interactive learning topic is challenging, while it is also very promising if we can develop some practical algorithms in this area.

implicit problems using matlab on clusters

CPU resource limits will now be enforced on Tensor. CPU usage will be monitored during the lifetime of each job and if the average CPU load exceeds the requested value of ncpus by 50% then the job will be automatically killed.

For example, if a job requests ncpus=1 but it actually uses eight cores then the job will end prematurely with a warning similar to the following:

PBS: job killed: ncpus 7.37 exceeded limit 1 (sum)

It is particularly easy to consume too many CPUs when using MATLAB because all versions of MATLAB since 2008a have multithreading enabled by default. Consequently, you may not be aware that your MATLAB job is using more than one CPU. Please consult the MATLAB documentation for further information on implicit multithreading.

how to compile matlab codes to standalone applciations

Sometimes, our matlab simulation may require a large amount of computation resources, which even causes hundreds of days for us to wait for the results. It seems unendurable and what we can do is just to give up the experiments. Is it true? No, we can find some solution to overcome this difficulty. Cluster! the same idea of Mapreduce. First, we dicompose the problem into subproblems, then we solve these subproblems using different processors in a cluster of computers. Finally, we combine the results of subproblems into one final solution to the original large-scale problem.

To use matlab on the cluster, the first problem we have to solve is to compile the matlab code into a standalone application. Because it not only speeds up the computation time but also is not limited by the restriction of maximum licenses for the matlab software we can use at the same time.

To compile the matlab, we have to configure the environment at the beginning. The command is

module add mcc

This command sets the lib path for matlab standalone executable. Without this setting, you may get the error saying that some shared library is missing. Next, we call

mcc -m -v main.m -a \sourcecode\

which not only compiles your main matlab codes but also the codes in the folder and subfolder of sourcecode.

Now we can talk about how to submit the executable file to clusters.

a project summary transfered from nyu Glimcher lab's website

Dopamine and Reinforcement Learning

---

Psychological and microeconomic theories of choice suggest that humans and animals must assign values to actions and objects in the world. These values can then be used to select the appropriate action or goal for a particular circumstance. Two major strands of research suggest that reinforcement learning is a mechanism that humans and animals use to learn these values. Classic behavioral studies of reinforcement learning in free choice environments have used the concurrent variable interval schedule introduced by Herrnstein in the late 1960's. In our lab we have extended this behavioral work and are now developing a replacement behavioral task better suited for neuroeconomic research.

Recent evidence has linked computational models of reinforcement learning (e.g. Sutton & Barto, 1998) originally derived from the psychological models of Bush & Mosteller and Rescorla & Wagner to the midbrain dopamine system. In particular, electrophysiological studies, suggest that dopamine neurons in the substantia nigra pars compacta (SNc) and ventral tegmental area (VTA) encode a reward prediction error (RPE) signal, the difference between experienced and anticipated reward.

Work done by Hannah Bayer in her thesis work in the Glimcher lab extended this research to show that dopamine neuron activity quantitatively encodes the predicted RPE signal (Bayer & Glimcher, 2004). Other labs have extended this research to show that the BOLD response in the Striatum (a dopamine target area) reflects a RPE signal as measured by fMRI in humans.

Further evidence for the encoding of values in the Striatum through reinforcement learning comes from electrophysiological recordings including the thesis work of Brian Lau in the Glimcher Lab. Brian demonstrated that both 'offer values' and 'chosen values' are represented in the Striatum. The time course of this neural encoding is compatible with possible roles in choice selection and the generation of RPE signals.

Reinforcement learning in monkeys with stimulation (Schafer).
Previous work (Schultz) indicates that SNc dopamine neurons encode a RPE when animals receive (or miss) a reward. We are extending this to actual decision tasks modeled after Hernnstein’s matching law (Herrnstein, 1961), where the animal chooses between two targets with different reward contingencies. We find that under choice conditions, dopamine firing rates are well predicted by the reinforcement learning models. Our current project causally tests the hypothesis that dopamine neurons are, in fact, encoding a RPE signal used in reinforcement learning. By actively stimulating dopamine neurons with pulses of current at the appropriate time during our choice task, we should cause the animal's predicted value of an option to increase and the animal’s behavior should change to reflect this.

Bandit task in humans (DeWitt, Dean).
The classic choice task developed by Herrnstein to study the 'Matching Law' has critical flaws when extended to the dynamic environments faced by humans and animals. We have developed a novel dynamic choice task based on the n-armed bandit problem widely studied in economics and computer science that overcomes these flaws. Importantly, we know the optimal strategy for our task on a choice-by-choice basis and this strategy is classic reinforcement learning! Our new task allows the measurement of the efficiency of reinforcement learning and to determine if humans and animals correctly trade-off the effect of noise against underlying changes in the environment as predicted by Bayesian theory.

Reinforcement learning in Parkinson's disease (Rutledge).
Parkinson's disease is characterized by a loss of dopamine neurons in the SNc and is associated with tremor, rigidity, and akinesia. The effect of this degeneration on reinforcement learning is unclear. To characterize human reinforcement learning we developed a task, adapted from our monkey choice task, in which subjects fish for crabs to earn money. By testing patients with Parkinson's disease both on and off dopaminergic medication, we find that reinforcement learning is modulated as predicted by theory. This project is a collaboration with Mark Gluck (Rutgers-Newark).

Methods for imaging dopamine areas in humans (DeWitt, Rutledge).
We are developing novel techniques to measure BOLD signals in dopamine projection and target areas in humans using Functional Magnetic Resonance Imaging (fMRI). We use the BOLD signal to provide an indirect measure of dopamine neural activity in humans. Unfortunately, current fMRI techniques make it difficult to accurately measure the midbrain dopamine areas and the orbito-frontal cortex (a major dopamine target area implicated in reinforcement learning). To better describe dopamine activity in choice tasks, we are developing imaging protocols to overcome measurement problems and functional and anatomical localizers to accurately and reliably find the dopamine areas. Our anatomical localizer uses an appropriate pulse sequence to image iron that accumulates in the dopamine areas as a byproduct of dopamine synthesis. Our functional localizer uses a classical conditioning task with primary rewards (juice) to identify dopamine areas. We have also developed a new method of image reconstruction using field map estimates to correct for signal dropout in the orbito-frontal cortex caused by magnetic field inhomogeneities near the air-filled sinuses.This project is a collaboration with Souheil Inati (Center for Brain Imaging, NYU).

An axiomatic model of dopamine function (Dean, Rutledge).
Although widely accepted, the dopamine RPE model has never been properly tested. We have developed a formal economic model which provides us with a number of testable axioms. We are collecting fMRI data using a task in which subjects choose between lotteries and observe the outcomes to win and lose real money. As expected, dopamine area activity is correlated with the predicted RPE signal. We are now testing whether dopamine area activity satisfies our economic axioms. This project is a collaboration with Mark Dean and Andrew Caplin (Economics, NYU).

2007-2008 remind the days we fall in love

From http://www.hellogrief.org/seven-years-later/comment-page-2/#comment-2521

Seven Years Later

By Hello Grief

• Email
• Print

By guest writer, Samantha Halle

In the days and weeks following my Dad’s death, countless people told me “it will get easier.” Now, seven years later, I can say that yes, in some ways it has. My Dad’s death is no longer one of the first things I remind myself of when I wake up, nor is it the last thing I think about before I fall asleep; it no longer consumes me.

But, even though it has been 2,655 days, I still miss him. I still have days and weeks when it’s just as painful as it was seven years ago, and I still have moments that make my head spin. There are several things, in particular, that almost always trigger one of these moments and force me to quite literally say hello to my grief. Here are the main five “little things” that get to me:

Telemarketers
Less than two weeks after my Dad died, I answered the phone only to hear a telemarketer struggling to pronounce my last name as he asked for my Dad. Feeling as if I had been slapped, I quickly hung up. In later calls, my response to the stinging words was a curt, defiant “NO.” About five years ago, one man dared to respond to my “NO” with, “he’s expecting my call. I spoke to him less than a week ago.” Most recently, my conversation with a persistent telemarketer went like this:

Telemarketer: “Is Mr. Hale there?”

Me: “No.”

Telemarketer: “When will he be in?”

Me: “He won’t. Please take our name off your list.”

Telemarketer: “Is there a better time I can call back to reach him?”

Me: “Nope.”

Telemarketer: “Uh, ok. I’ll try back another time.”

Me: “Good luck.”

Though telemarketers don’t bother me as much as they initially did, they still get to me sometimes. They serve as just one more reminder that my Dad is gone.

Questions about Family
There are frequently questions when you meet someone new, and based on the majority of my experiences, these questions are typically asked by curious, or trying-to-be-polite, adults. Many adults I babysit for will ask what my parents do for a living, and I always hesitate to consider my answer.

I typically respond by stating what my Mom does. Still, many adults will complete my answer with “…and your Dad?”

Several years ago I would neglect to mention that he had died and would simply say what he used to do. Now, if necessary, I will quickly add “my Dad died when I was 11.”

Of course, the instant I release these words into the air, I see the change on their face. They quickly try to smooth their stunned expression and mutter an “I’m sorry.” Then, in an almost ironic way, I console them, letting them know that it’s OK—I’m OK.

Things that mean something more to you
There have been countless occasions when I’m watching a movie or TV show, or listening to a song or story with a friend, when something hits me. A line or situation sticks out, reminding me of my Dad in some way. Suddenly something’s different; there’s a pang of sadness, a feeling of nostalgia, or a flood of bittersweet sentiment.

Sometimes this moment is brief and I bounce back immediately. Other times, I feel the tears rushing to my eyes and am forced to actively remain composed.

Accomplishments
There’s something incredibly bittersweet about accomplishments, knowing that my Dad’s not here to enjoy them with me.

My Dad was the proud, brag-about-your-kids type of guy. He was front and center at every play and recital, and cheering at the end of the pool during each and every swim meet. Now, if I win an award or have something major happen in my life, I have a moment of longing, wishing he could be here to see what I’ve done and know the person I’ve become.

Time
Hands down, one of the hardest things that has come with losing my Dad is the occasional realization of how much time has passed. Birthdays, holidays, and other milestones are all reminders.

There are days when I feel like it was just yesterday that he died, but other times, I feel as if it has been a lifetime and I can no longer imagine my life with him in it.

There are moments when I must consciously think about how long it has been since he died; it’s as if having him here was a past life of mine—a movie that I’ve watched countless times and memorized but never actually lived. There are times when I realize that I’m slowly forgetting things I swore I never would and it scares me. So, I make a concerted effort to replay poignant moments in my mind.

Many people who have not lost someone mistakenly believe that death is something you will “get over.” However, the truth is, I still hurt. Seven years later, it’s not a constant, overwhelming, consuming grief, but the little things, within which grief hides, that hit me when I least expect it.

Python: Dynamic Typing

Note: Lines beginning with ">>>" and "..." indicate input to Python (these are the default prompts of the interactive interpreter). Everything else is output from Python.

The following example demonstrates how a function can examine its own arguments and do different things depending on their types. The names of the types ("IntType" etc.) are defined in a module called "types", so we have to import them.

>>> from types import * >>> >>> def what (x): ... if type(x) == IntType: ... print "This is an int." ... else: ... print "This is something else." ... >>> what(4) This is an int. >>> >>> what("4") This is something else.

This kind of dynamic typing has advantages as well as disadvantages.

The advantage is clearly that it gives a lot of freedom to the programmer and enables him to do things that wouldn't be possible otherwise. For example, you can write functions to which you can pass an integer as well as a string or a list or a dictionary or whatever else, and it will be able to transparently handle all of them in appropriate ways (or throw an exception if it cannot handle the type). You can do things like that in other languages, too, but usually you have to resort to (ab)use things like pointers, references or typecasts, which opens holes for programming errors, and it's just plain ugly.

The obvious disadvantage of dynamic typing is that the compiler cannot perform complete typechecks at compile-time, therefore it is possible that bugs creep in that are hard to find. On the other hand, Python's capabilities to handle types and runtime-errors (exceptions) in a very natural and convenient way prevents such problems most of the time. Furthermore, there are several other ways to assist in writing correct code, such as the assert statement and the doctest module for automated regression tests.

By the way, instead of importing the types module, you can compare the type of a variable with the type of a known constant. So, the above function could be rewritten like this:

>>> def what (x): ... if type(x) == type(1): ... print "This is an int." ... else: ... print "This is something else."

Beginning at Python 2.3, the names of built-in functions that create values of a specific type (int, float, str, list, tuple etc.) can be used as types, too. So the above conditional can be simply rewritten like this:

... if type(x) == int:

---

Here is a more complicated example. It defines a so-called dictionary (in other languages, these are called associative arrays or hashes). The keys of the dictionary are types, and the values are lambda functions that handle two arguments of that type.

By the way, you can use anything as keys in a dictionary, even complicated structured types if you want, as long as they are "immutable" (i.e. they cannot be changed, for obvious reasons).

>>> jobs = { ... IntType: lambda x, y: x**2 + y, ... StringType: lambda x, y: y.join(x.split()) ... ListType: lambda x, y: [min(x), max(x), min(y), max(y)] ... } >>> >>> def something(a, b): ... if type(a) != type(b): ... raise "Arguments not of the same type" ... if not jobs.has_key(type(a)): ... raise "Don't know how to handle this type" ... return jobs[type(a)](a, b) ... >>> print something(4, 5) 21 >>> >>> print something("This is so useless.", " ... ") This ... is ... so ... useless. >>> >>> print something([5, 3, 11, 9, 7], [20, 16, 8, 24, 14, 18]) [3, 11, 8, 24] >>> >>> print something(3, "7") Traceback (most recent call last): File "<stdin>", line 1, in ? File "<stdin>", line 3, in something Arguments not of the same type

This is not a useful real-world example, of course, but it demonstrates some of the possibilities. Note that Python does not require a large amount of cryptic syntax. You can code powerful tasks in simple, readable statements.