On the third floor of Citigroup’s Manhattan headquarters, at the far end of a trading floor overlooking the Hudson River, Young Kang, Citi’s global head of algorithmic products, leans over a terminal and monitors the progress of a canny and powerful beast named Dagger. Bred and trained in secret by Citi’s financial engineers, Dagger can stalk through more than 20 markets, public and otherwise—hunting for anomalies, buying and selling, prowling through mountains of historical data—all at the behest of Citi’s clients. Amid the trading-floor din, Dagger fulfills its duties in flickering silence, with a speed and acuity no human can match.

“It’s self-learning,” Kang says. “The numbers keep updating, the strategy keeps adjusting itself. It gets smarter.”

And it makes a lot of money. Algorithms like Dagger can exploit the smallest inefficiencies in the market. They can parse trades in millionths of a second. Some species can detect other algos embarking on predictable trading strategies, and ruthlessly adjust their techniques. They’re growing ever more complex, subtle, and sophisticated. And as they become more popular, they’re creating some serious headaches for regulators.

By some estimates, algorithms now trigger 70 percent of all trades in U.S. equities. The speed and volume of everyday trading have propelled the market into a new and esoteric dimension, and rendered traders in the pits largely obsolete. Average daily share volume on the New York Stock Exchange increased by 181 percent between 2005 and 2009, while the time required to execute a trade on its electronic systems dropped to 650 microseconds.

Such changes have a lot of people worried, including the Securities and Exchange Commission. It released a wide-ranging paper earlier this year seeking suggestions on how to restructure the entire equity market, and created a Division of Risk, Strategy, and Financial Innovation in part to help monitor new technologies. A market collapse in early May—in which automated-trading systems exacerbated a sell-off that drove the Dow down more than 900 points in less than an hour, before it quickly recovered—gave two worries new public salience: that the proprietors of these algos may not be in full control of their creations, and that the strategies they pursue are, in some cases, fundamentally warping the financial markets.

In January, the NYSE fined Credit Suisse $150,000 for “failing to adequately supervise the development, deployment, and operation of a proprietary algorithm.” The fine was a pittance, but more troubling was that the bank didn’t even know that its malfunctioning algo (which sent hundreds of thousands of cancel-and-replace requests for orders that hadn’t been made) had crippled some of the NYSE’s trading stations until regulators called them the next day. This spring, a newsletter from the Federal Reserve Bank of Chicago warned: “Although algorithmic trading errors have occurred, we likely have not yet seen the full breadth, magnitude, and speed with which they can be generated. Furthermore, many such errors may be hidden from public view.”

Bernard Donefer, a finance professor at Baruch College and the author of a study in the most recent Journal of Trading called “Algos Gone Wild,” contends that the speed of these equations, and their ability to reach so many markets simultaneously, could turn even a minor coding error into a spiraling disaster. “Another 1987,” he told me, referring to the epic crash caused in part by simpler automated-trading schemes. This view puts Donefer in the minority in the financial community, which tends to have more faith in firms’ internal risk controls. But he thinks that without better regulation, more algo-gone-wild scenarios are inevitable. He notes that while controls at big firms, like Citi, are generally exemplary, second- and third-tier firms present a graver risk.

The SEC wants to hire a lot more staffers, both for its new risk division and for its trading division, and it is considering new methods of tracking algorithmic trades; Donefer and others have suggested a tagging system for the biggest traders, which the SEC says is on the table. The commission also may soon outlaw a practice called “naked access,” in which some broker-dealers offer their clients direct access to exchanges—allowing them to potentially bypass risk controls—in pursuit of faster trading.

A more widespread worry, now getting increased attention from regulators and Congress, is a strategy known as high-frequency trading. Employers of this technique apply algorithms and other automated technology, along with real-time market data, to buy and sell so quickly (in microseconds) and in such quantities (millions of trades a day), that they engorge themselves on penny differentials in prices. These traders argue that they supply the market with needed liquidity and tighter spreads. Regulators tend to agree, for the most part; free markets have always rewarded better information, speed, and creativity. But this technology unloads on such a massive scale, and so quickly, that they fear it could feed a dangerous and self-reinforcing volatility.

At least a few high-frequency traders have learned to make a killing by detecting the more simplistic algo strategies deployed by basic pension funds and mutual funds, buying the next stock the funds plan to buy, and then selling it to them at a higher price. This may not be illegal, but it’s almost certainly unfair to the funds’ investors. “It is increasingly clear that there are quite a number of high-frequency bandits in the high- frequency-trading community who pump up volume statistics, front-run investor orders, increase transaction costs, and hurt real liquidity,” David Weild, an adviser at Grant Thornton and a former vice chairman of Nasdaq, told me. *

These changes in trading technology raise a more fundamental question: If the majority of trades racing back and forth are simply lines of code swapping with other lines of code, moved by indicators obscure to even the mortal authors of the algorithms themselves, what exactly is the financial market? “The market structure’s totally changed, and it’s distorted what we do,” says Joe Saluzzi, the co-head of equities trading at Themis Trading and a vocal opponent of some high-frequency strategies. “The machine thinks for itself.”

Correction: The print version of this article incorrectly stated that David Weild was vice president of NASDAQ. He was vice chairman.

Its WOPR, or Joshua: "Would You Like To Play A Game?"

A cognitive bias is something that our minds commonly do to distort our own view of reality. Here are the 26 most studied and widely accepted cognitive biases.

1. Bandwagon effect – the tendency to do (or believe) things because many other people do (or believe) the same. Related to groupthink, herd behaviour, and manias. Carl Jung pioneered the idea of the collective unconscious which is considered by Jungian psychologists to be responsible for this cognitive bias.
2. Bias blind spot – the tendency not to compensate for one’s own cognitive biases.
3. Choice-supportive bias – the tendency to remember one’s choices as better than they actually were.
4. Confirmation bias – the tendency to search for or interpret information in a way that confirms one’s preconceptions.
5. Congruence bias – the tendency to test hypotheses exclusively through direct testing.
6. Contrast effect – the enhancement or diminishment of a weight or other measurement when compared with recently observed contrasting object.
7. Déformation professionnelle – the tendency to look at things according to the conventions of one’s own profession, forgetting any broader point of view.
8. Disconfirmation bias – the tendency for people to extend critical scrutiny to information which contradicts their prior beliefs and uncritically accept information that is congruent with their prior beliefs.
9. Endowment effect – the tendency for people to value something more as soon as they own it.
10. Focusing effect – prediction bias occurring when people place too much importance on one aspect of an event; causes error in accurately predicting the utility of a future outcome.
11. Hyperbolic discounting – the tendency for people to have a stronger preference for more immediate payoffs relative to later payoffs, the closer to the present both payoffs are.
12. Illusion of control – the tendency for human beings to believe they can control or at least influence outcomes which they clearly cannot.
13. Impact bias – the tendency for people to overestimate the length or the intensity of the impact of future feeling states.
14. Information bias – the tendency to seek information even when it cannot affect action.
15. Loss aversion – the tendency for people to strongly prefer avoiding losses over acquiring gains (see also sunk cost effects)
16. Neglect of probability – the tendency to completely disregard probability when making a decision under uncertainty.
17. Mere exposure effect – the tendency for people to express undue liking for things merely because they are familiar with them.
18. Omission bias – The tendency to judge harmful actions as worse, or less moral, than equally harmful omissions (inactions).
19. Outcome bias – the tendency to judge a decision by its eventual outcome instead of based on the quality of the decision at the time it was made.
20. Planning fallacy – the tendency to underestimate task-completion times.
21. Post-purchase rationalization – the tendency to persuade oneself through rational argument that a purchase was a good value.
22. Pseudocertainty effect – the tendency to make risk-averse choices if the expected outcome is positive, but make risk-seeking choices to avoid negative outcomes.
23. Selective perception – the tendency for expectations to affect perception.
24. Status quo bias – the tendency for people to like things to stay relatively the same.
25. Von Restorff effect – the tendency for an item that “stands out like a sore thumb” to be more likely to be remembered than other items.
26. Zero-risk bias – preference for reducing a small risk to zero over a greater reduction in a larger risk.

Oh and, by the way, you’ll never be able to truly gauge any of the biases you might be operating under since it’s not possible to accurately observe a system you’re part of. Now, get out there and delude yourself!

Complete list of cognitive biases – Wikipedia

______________________________________

Its not often I get to apply some of my 'formal' education within my current excuse for a day job...

So, I was writing an article about the unpredictability of mortgage rates (and financial markets in general).
In the process I researched some of the more popular pundits to discover their 'predictions' were consistently wrong and/or self-serving...or biased to benefit them or their sponsor...which triggered a memory from a psych class in college about Carl Jung and the 'Collective Unconscious' and cognitive biases in general.

So I 'Googled' around some more and found the above article with 26 solid examples of cognitive biases.

I know you can't truly self-diagnose, but I just know I suffer from Planning Fallacy :-)

Which reality do you think you may be distorting?

Crowdsourcing a billion dollar class action lawsuit against Apple and AT&T!

Do you have an iPhone? Then you know it's the best portable computer ever made, while at the same time being the worst. phone. ever. because it drops calls all the time!

Here's what you can do: upload log files of dropped calls for your phone, see how many calls you dropped, where, and when, and then see how they compare with your friends!

 

Then, when we have enough data, we're going to file a class action lawsuit on behalf of all our users, run Apple and AT&T through the ringer, and you can get a slice of the action! Don't get mad, get even!

 

-Wonder if you can make an App for that? :-)

Via online.wsj.com

By EMILY STEEL And JESSICA E. VASCELLARO

Facebook, MySpace and several other social-networking sites have been sending data to advertising companies that could be used to find consumers' names and other personal details, despite promises they don't share such information without consent.

The practice, which most of the companies defended, sends user names or ID numbers tied to personal profiles being viewed when users click on ads. After questions were raised by The Wall Street Journal, Facebook and MySpace moved to make changes. By Thursday morning Facebook had rewritten some of the offending computer code.

Advertising companies are receiving information that could be used to look up individual profiles, which, depending on the site and the information a user has made public, include such things as a person's real name, age, hometown and occupation.

Several large advertising companies identified by the Journal as receiving the data, including Google Inc.'s DoubleClick and Yahoo Inc.'s Right Media, said they were unaware of the data being sent to them from the social-networking sites, and said they haven't made use of it.

Across the Web, it's common for advertisers to receive the address of the page from which a user clicked on an ad. Usually, they receive nothing more about the user than an unintelligible string of letters and numbers that can't be traced back to an individual. With social networking sites, however, those addresses typically include user names that could direct advertisers back to a profile page full of personal information. In some cases, user names are people's real names.

Most social networks haven't bothered to obscure user names or ID numbers from their Web addresses, said Craig Wills, a professor of computer science at Worcester Polytechnic Institute, who has studied the issue.

The sites may have been breaching their own privacy policies as well as industry standards, which say sites shouldn't share and advertisers shouldn't collect personally identifiable information without users' permission. Those policies have been put forward by advertising and Internet companies in arguments against the need for government regulation.

View Full Image

 

Bloomberg News

Facebook, MySpace and several other social-networking sites gave advertising companies information that could be used to look up individual profiles, which, depending on the site and the information a user has made public, include such things as a person's real name, age, hometown and occupation. Above, Facebook's headquarters in Palo Alto, Calif.

 

The problem comes as social networking sites—and in particular Facebook—face increasing scrutiny over their privacy practices from consumers, privacy advocates and lawmakers.

At the same time, lawmakers are preparing legislation to govern websites' tactics for collecting information about consumers, and the way that information is used to target ads.

In addition to Facebook and MySpace, LiveJournal, Hi5, Xanga and Digg also sent advertising companies the user name or ID number of the page being visited. (MySpace is owned by News Corp., which also owns The Wall Street Journal.) Twitter—which doesn't have ads on profile pages—also was found to pass Web addresses including user names of profiles being visited on Twitter.com when users clicked other links on the profiles.

For most social-networking sites, the data identified the profile being viewed but not necessarily the person who clicked on the ad or link. But Facebook went further than other sites, in some cases signaling which user name or ID was clicking on the ad as well as the user name or ID of the page being viewed. By seeing what ads a user clicked on, an advertiser could tell something about a user's interests.

Journal Community

• Vote: How would you grade Facebook's handling of user's privacy?

Ben Edelman, an assistant professor at Harvard Business School who studies Internet advertising, reviewed the computer code on the seven sites at the request of the Journal.

"If you are looking at your profile page and you click on an ad, you are telling that advertiser who you are," he said of how Facebook operated, if a user had clicked through a specific path, before the fix. Mr. Edelman said he had sent a letter on Thursday to the Federal Trade Commission asking them to investigate Facebook's practices specifically.

The sharing of users' personally identifiable data was first flagged in a paper by researchers at AT&T Labs and Worcester Polytechnic Institute last August. The paper, which drew little attention at the time, evaluated practices at 12 social networking sites including Facebook, Twitter and MySpace and found multiple ways that outside companies could access user data.

The researchers said in an interview they had contacted the sites, which some sites confirmed. But nine months later, the issue still exists.

The issue is particularly significant for Facebook on two fronts: the company has been pushing users to make more of their personal information public and the site requires users to use their actual names when registering on the site.

A Facebook spokesman acknowledged it has been passing data to ad companies that could allow them to tell if a particular user was clicking an ad. After being contacted by the Journal, Facebook said it changed its software to eliminate the identifying code tied to the user from being transmitted.

"We were recently made aware of one case where if a user takes a specific route on the site, advertisers may see that they clicked on their own profile and then clicked on an ad," the Facebook spokesman said. "We fixed this case as soon as we heard about it."

Facebook said its practices are now consistent with how advertising works across the Web. The company passes the "user ID of the page but not the person who clicked on the ad," the company spokesman said. "We don't consider this personally identifiable information and our policy does not allow advertisers to collect user information without the user's consent."

The company said it also has been testing changing the formatting for the text it shares with advertisers so that it doesn't pass through any user names or IDs.

Privacy Problem

Internet sites that share information that could be tied to individual profiles. See graphic

 

MySpace, Hi5, Digg, Xanga and Live Journal said they don't consider their user names or ID numbers to be personally identifiable, because unlike Facebook, consumers are not required to submit their real names when signing up for an account. They also said since they are passing along the user name of the page the ad is on, not for the person clicking on the ad, there is nothing advertisers can do with the data beyond seeing on what page their ad appeared.

MySpace said in a statement it is only sharing the ID name users create for the site, which permits access only to the information that a user makes publicly available on the site.

Nevertheless, a MySpace spokeswoman said the site is "currently implementing a methodology that will obfuscate the 'FriendID' in any URL that is passed along to advertisers."

A Twitter spokeswoman said passing along the Web address happens when people click a link from any Web page. "This is just how the Internet and browsers work," she said.

Although Digg said it masks a user's name when they click on an ad and scrambles data before sharing with outside advertising companies, the site does pass along user names to ad companies when a user visits a profile page. "It's the information about the page that you are visiting, not you as a visitor," said Chas Edwards, Digg's chief revenue officer.

The advertising companies say they don't control the information a website chooses to send them. "Google doesn't seek in any way to make any use of any user names or IDs that their URLs may contain," a Google spokesman said in a statement.

"We prohibit clients from sending personally identifiably information to us," said Anne Toth, Yahoo's head of privacy. "We have told them. 'We don't want it. You shouldn't be sending it to us. If it happens to be there, we are not looking for it."

Write to Emily Steel at emily.steel@wsj.com and Jessica E. Vascellaro at jessica.vascellaro@wsj.com

You're not a real Social Media Guru until...

 

Potentially disruptive news for Realtors: Augmented reality browser Junaio has partnered with real estate search engines ROFO and HotPads—commercial and residential, respectively—to make finding the perfect space easy, sans wily brokers and landlords.

Just hold your phone up to a city block or neighborhood, and the app shows what spaces are available for sale or rent, as well as square footage, pictures, contact information, and price listings. Having knowledge of neighborhood pricing and vacancies before meeting with brokers? That’s like a secret weapon filled with bargaining power.

“Our goal with our Web site was to make a housing search easier. We’ve expanded that capability from a computer screen to a smartphone screen,” says Douglas Pope, HotPads co-founder. “Our users can conduct their search while they’re walking around the neighborhood or even riding in a car.”

The Junaio app—which is free, and available for the iPhone and soon, Android—has access to the real estate listings databases of both ROFO and HotPads, guaranteeing a pretty comprehensive look at available spaces, potentially saving apartment and office hunters from some serious headaches.

 

The pursuit of machine intelligence means we have to come up with ways to communicate with our computers in a way both entities can understand. But while computers process verbal commands in a straightforward fashion, humans tend to use more sophisticated speech forms, employing slang or symbols to convey an idea. So an Israeli research team has developed a machine algorithm that can recognize sarcasm.

SASI, a Semi-supervised Algorithm for Sarcasm Identification, can recognize sarcastic sentences in product reviews online with pretty astounding 77 percent precision. To create such an algorithm, the team scanned 66,000 Amazon.com product reviews, with three different human annotators tagging sentences for sarcasm. The team then identified certain sarcastic patterns that emerged in the reviews and created a classification algorithm that puts each statement into a sarcastic class.

The algorithms were then trained on that seed set of 80 sentences from the collection of reviews. These annotated sentences helped the algorithm learn what sorts of words and patterns distinguish sarcastic remarks – those that mean the opposite of what they literally convey, or that convey a sentiment inconsistent with the literal reading.

 

They then turned the algorithm loose on an evaluation set. Pattern evaluation efficiency scored accurately 81 percent of the time, while the overall precision of the pattern recognition/sarcasm categorizing algorithm was accurate in 77 percent of instances. Not bad for a computer’s first shot at interpreting the human sense of humor.

This isn’t all just so your Roomba gets the joke when you tell it it sucks. Computer programs that can recognize sarcastic statements could generate better personalized content and make better recommendations to human users by not mistaking a product review titled “keep your receipt” with a sound piece of online shopping advice. It could also benefit opinion-mining systems that troll the Web trying to measure public sentiment about a product or idea.

[Slashdot]

Sweet. My snark is less likely to be lost on an unsuspecting person or product. This company should consult with my mom, with her 30++ years experience in dealing with the model child of sarcasm...they could also use her 'bullshit detector' that she wielded on me and my friends with great effectiveness. 

-JX

Via: Inman.com- "Real estate data management company Threewide Corp. has released a report to help brokerages compare 55 listing syndication services, the company announced Wednesday."

Nice tool...