Doing Bad Data Science

I finished reading through Statistics Done Wrong e-book (free) which suggested the perils of bad statistical analysis.  The author goes on to ironically further suggest statistical evidence of rampant bad analytics in academia, as evidenced via fact-checking of published academic papers.  Maybe this is a sign that the author was pretty effective a teacher at teaching data skepticism?

After months of procrastinating, I’ve finally gotten around to learning the Python libraries numpy and pandas within the iPython environment, all of which wrapped in the Enthought Canopy IDE’s virtual environment. All of this is eerily similar to using R inside RStudio, minus the virtualization. I personally prefer doing all sorts of general computering in Python and I love the %timeit magic command in Enthought Canopy, which allows me to profile single lines of code while staying in the ipython environment. I’d say that knowing Python is merely psychological comfort because as numpy rhymes with the batteries-included data structures, pandas is more of a stranger.

I’m currently flirting with the idea of technical analysis, but drawing conclusions from asymmetric time series data just seems like something that would not work in theory. And in practice that data comes in simply too slow to compete with the professionals. Despite my own doubts, I’ve been enjoying the Poor Man’s High Speed Trading Platform, Quantopian. It has the neatest little forum that incorporates source code, message boards, and the ability to immediately run or modify the discussed code. I also found out about QuantConnect today, which is the C# alternative to Quantopian’s Python. Currently Quantopian is the livelier forum and has already begun beta testing of Interactive Broker integration, but I’ll be keeping an eye on both!


Sweet Potato Congee Recipe

Being down this weekend with a cold, I decided to make myself a congee.  My recipe turned out well, so saving it here!

– 1 teaspoon of ginger
– 1 teaspoon of vegetable oil
– 2 cups of Trader Joes Brown Rice Medley (with daikon radish seeds!)
– 4 cups of water
– 2 small white sweet potatoes

Chop up the ginger into tiny bits. Peel and cube the sweet potatoes. I usually shortcut by boiling water in an electric kettle then add to my 4 qt dutch oven.  Add the rest into the dutch oven at medium or medium-high temp until boiling.  Then reduce to simmer and cover for 1-2 hrs until the water is slightly porridge-y but the rice itself still has shape.  Serve with typical sides like spicy baked tofu, bamboo shoots in chili oil, pickled cucumber, etc.

A Cartoonist’s Advice

In honor of Labor Day, a comic by Bill Watterson


Rachel Botsman: The Case for Collaborative Consumption

I cheer any chance I get to see an articulate, intelligent woman wishing to change the world. However, there is one downside of collaborative consumption which we have already experienced in world history, and this grand social experiment has failed in many places including both Europe and Asia.  Botsman’s over-enthusiasm for sharing, for example, a land owner sharing land with a farmer in exchange for food services, is veering dangerously into the backward territory of feudalism.

Feudalism was a set of legal and military customs in medieval Europe that flourished between the 9th and 15th centuries, which, broadly defined, was a system for structuring society around relationships derived from the holding of land in exchange for service or labour.


I agree that a sharing economy should be encouraged without endangering the concept of ownership.  Niall Fergusson speaks of “6 killer apps” that allowed Western Civilization to achieve imperium over the rest of the world:

  • Competition
  • Scientific Revolution
  • Property Rights
  • Modern Medicine
  • Consumer Society
  • The Work Ethic

It appears that Fergusson argues that 1/3 of of the success of the West was due to ownership.  I believe that, to some extent, property rights and a consumer society is necessary for individuals to feel ownership in the system or “have some skin in the game”.   Land prices may be skyrocketing, but all paths do not have to lead to feudalism.


Post-scarcity Society or the Future of Work

Having recently read the “Race Against The Machine: How the Digital Revolution is Accelerating Innovation, Driving Productivity, and Irreversibly Transforming Employment and the Economy“, I found this work to provide a quick and concise job of outlining how technology today is leading us towards a post-scarcity future.  I’m seeing an uptrend in others taking note of this alarming trend and adding their own perspectives into the mix.

From “THE WASTEFULNESS OF AUTOMATION” where Frances Coppola expounds on not only the wastefulness of a fully automated, future society but the imminent collapse of society that it could usher in:

The fact is that robots are brilliant at supply, but they don’t create demand. Only humans create demand – and if the majority of humans are so poor that they can only afford basic essentials, the economy will be constrained by lack of demand, not lack of supply. There would be no scarcity of products, at least to start with….but there would be scarcity of the means to obtain them.

But the short-sighted strategy of forcing down wages to prop up profits is not the only problem. As Tomas Hirst notes, traditional “middle class” skilled production and office jobs are disappearing, but there is relative growth of low-skill, low-pay jobs, mostly insecure, part-time and short-term. These jobs are increasing because the cost of employing people to do them is lower than the cost (at present) of automating them. If the future is that the majority of people will do unskilled, insecure jobs for very low wages, then this amounts to a shocking waste of human capital. And if the more distant future is that even these jobs will eventually be automated, and working for a living will become the privilege of a few, then it is an even bigger waste.We have the most educated workforce in history, but the majority of them will have no opportunity to use their skills in satisfying and well-remunerated work.

From “You are not an Artisan“, the writer argues the opposite claiming that work is endless but the special or artisan aspect of work will not be in short supply:

In other words, we’re more afraid of machines taking away our social status than our jobs. This might seem like an obvious point. After all, most status-conscious people have strong feelings about what work is “beneath” them, but with machines in the picture, the point gets considerably more subtle.

Wired has a much more optimistic idea of the future as well, ignoring the possibility that millions will be displaced without opportunity to support themselves, simply focusing on information workers who already have a close relationship with automation.  This makes for an optimistic future for workers in information technology.

In the coming years our relationships with robots will become ever more complex. But already a recurring pattern is emerging. No matter what your current job or your salary, you will progress through these Seven Stages of Robot Replacement, again and again:

    • 1. A robot/computer cannot possibly do the tasks I do.[Later.]
    • 2. OK, it can do a lot of them, but it can’t do everything I do.[Later.]
    • 3. OK, it can do everything I do, except it needs me when it breaks down, which is often.[Later.]
    • 4. OK, it operates flawlessly on routine stuff, but I need to train it for new tasks.[Later.]
    • 5. OK, it can have my old boring job, because it’s obvious that was not a job that humans were meant to do.[Later.]
    • 6. Wow, now that robots are doing my old job, my new job is much more fun and pays more![Later.]
    • 7. I am so glad a robot/computer cannot possibly do what I do now.

This is not a race against the machines. If we race against them, we lose. This is a race with the machines. You’ll be paid in the future based on how well you work with robots.

The Atlantic, in Robots and the Future of Unemployment, expresses the current transformation as an echo of the past:

Transformation of the skill level of that much labor is going to have large destabilizing effects. The last time we had such a transition of the skilling of labor we’d have to look at the Industrial Revolution in Europe. There we had a transformation from an artisan class into a deskilled, more productive class. This, from an economists point-of-view (but certainly not everyone’s), was a huge win for unskilled labor. This new change in the skilling of labor is not what is happening now. Now the economy is taking the labor of an unskilled class and replacing it with machines. In order for someone to take advantage of the new employment opportunities opening up it will requires a peculiar new type of skilling – the ability to create and maintain the new efficiency machines.

Kurt Vonnegut also explores this concept similarly in a utopia-dystopia novel Player Piano.

Yet another post on this subject from the Washington Post suggests a solution:

Bridging the educational divide, to help lower-income students succeed in the robot-proof workforce, is a huge undertaking, Murnane and Levy concede. Murnane called it a decades-long challenge — but one that, ironically, could gain urgency among policymakers if the pace of technological advancement accelerates and more people find their jobs jeopardized by automation.

“One of the things that operates right now is there are a chunk of people who are doing just fine, because they’re not threatened by technology,” Murnane said. “Once you see more and more people threatened, that really changes the political calculation.”

Confusing Characters Juxtaposed – Basic

















Progress Update:

Last week, I decided to try my Chinese literacy chops on the Chinese Engadget blog, with disheartening results of only about 10% recognition!  One of the typically short articles would take me an hour to slog through with the help of my ZhongWen Chrome Plug-in.  I had found that diving into these articles–cold–was not going to get me anywhere fast, even if the articles actually covered topics of interest to me (rather than the typical childrens’ mundane topics).  After having spent the last few days going through flash cards for the basic ~1500 characters, I’m finding a much improved reading speed of 15 minutes of a sample article, mostly due to being able to recognize these basic characters.  I can honestly say I’ve never reached this level of literacy in my life (discounting the childrens textbooks that had always had the aid of bopomofo pronunciation next to it)!  And sadly, it only took a week to get up to this speed.  I am trying not to dwell on the sad fact that I could have had decades, at leisure, to slowly build up my “medium” level literacy.   By the end of next week, I should have sufficiently committed the basic characters to mid/long term memory.  Then, after two decades of active aversion to this subject, I’ll finally be leveling up!

Assorted Links – News from Ancient Rome

Tale of Glorious Art and Not So Glorious Thieves – Etruscan Artifacts Looted by Amateurs Are Prize Objects

Futuo! How the Romans Swore

Ancient Roman Concrete Recipe
– Only lost for thousands of years thanks to the collapse of empire! But this sturdy material should withstand the sorry state of Roman architectural preservation?

Finally, an immersive 3rd person action adventure set in ancient Rome:

Wisdom to strive towards for those big and small

心胸 大 的 人談論 理想,心胸 小 的人 論 人 是非。


Great minds discuss ideas, average minds discuss events, small minds discuss people.  — Eleanor Roosevelt


I came across this sentence as I was learning Chinese and liked it.  It seems appropriate on the 4th to also quote a first lady saying something similar.  It seems also appropriate on this day celebrating our independence to remind that modern politics seems to be a weird amalgam of big minds and small minds, of those who discuss ideas and of those who discuss politicians.

The Race of Our Lives — Jeremy Grantham

The collapse of civilizations is a gripping and resonant topic for many of us and one that has attracted many scholars over the years. They see many possible contributing factors to the collapse of previous civilizations, the evidence pieced together shard by shard from civilizations that often left few records. But some themes reoccur in the scholars’ work: geographic locations that had misfortune in the availability of useful animal and vegetable life, soil, water, and a source of energy; mismanagement in the overuse and depletion of resources, especially forests, soil, and water; the  lack of a safety margin or storage against inevitable droughts and famines; overexpansion and costly unnecessary wars; sometimes a failure of moral spirit as the pioneering toughness and willingness to sacrifi ce gave way to softer and more cynical ways; increasing complexity of a growing empire that became by degree too expensive in human costs and in the use of limited resources to justify the effort, until the taxes and other demands on ordinary citizens became unbearable, so that an empire, pushed beyond sustainable limits, became vulnerable to even modest shocks  that could in earlier days have been easily withstood. Probably the greatest agreement among scholars, though, is that the failing civilizations suffered from growing hubris and overconfidence: the belief that their capabilities after many earlier tests would always rise to the occasion and that growing signs of weakness could be ignored as pessimistic. After all, after 200 or even 500 years, many other dangers had been warned of yet always they had persevered. Until finally they did not.


Another theory for civilization evolution

At the same press conference Matthew Konfirst, from the Ohio State University, discussed the disappearance of the Sumerian civilization and their language. In the cradle of civilization in the Middle East — about 4,000 years ago — there were two major language families, he said. One eventually became Hebrew and Arabic and other familiar languages, whereas the other family, Sumerian, stopped evolving when that civilization collapsed, a linguistic dead end. That collapse coincided with a drought that lasted hundreds of years, he said.

—   Climate Change and the Rise and Fall of Civilizations

I’ve been studying several theories to explain the rise and fall of civilizations. So far, I’m starting to form the opinion that many were due to or the consequence of climate change. It is nice to know that, based on this article, some researchers are also attempting to corroborate this theory!


Previous Older Entries

Greatest hits

  • 1,395 hits