Good relationships, great relationships

A friend of mine and I talked about relationships the other day.  I realized that there are good relationships, and there are great relationships.

You may have been lucky to be in a good relationship.  I think that these can be characterized with an absence of guilt.  I've been in relationships that have been marked by embedded guilt (even if it's implicit, "because you don't want to hurt the other person") and in those that have not and the difference is enormous.

Most, if not all guilt comes about because we are fundamentally insecure creatures.  We don't like to be criticized, we are cautious by nature, we like to stand on firm ground that nobody questions and wear a thin skin that nobody pinches.  In relationships, this is particularly important because we are by definition sharing something with another person; insecurity only stands in the way of this.  This is what makes us want to present ourselves in a better light; as a result, we're not ourselves -- we are what we think others would like us to be if they were us.  In many relationships, this facade is kept up for a long time (again, not always deliberately, and often implicitly) but the tension remains (because it's still just a facade).  This tensions causes those relationships to collapse.  So if you've been privileged to be in a good relationship, this would not have been a problem.

Great relationships are very few and far between.  A lot of people may think they've been in great relationship but that's naive.  I think great relationships are marked by our willingness (not just "being OK") to relinquish control and comfort with sharing everything with the other person.  My friend told me he thought he had a great relationship but later it turned out that he would never want his significant other to get access to his email archive.  Why not--if you're relinquishing control and comfortable with sharing everything with the other person?  Well, my friend said, he may have said something about himself, or that significant other, that he wouldn't like the significant other to know.  Why not?  In a truly great relationship, the significant other would understand.

What are great relationships?  Why are they better than living alone?  And why are they so special?  I think the only thing that's better than being comfortable with yourself is being able to share this comfort with another human being.  There's something about human interactions that makes us go a little crazy if we have to keep too much to ourselves.  As my friend said, so much is happening in his head that he is constantly wondering if he's the crazy one (surrounded by normal people).  Obviously, though, everyone thinks they are crazy and everyone else is sane, so we can sleep at night, but the paradox remains: with so little feedback we suffer.

And since in a great relationship, you have the opportunity to share with another human being, there is no need to cheat (what for? to share something different with a different other human being?).  There is no need to lie (unless in circumstances when you would lie to yourself).  In a way, the only danger in a great relationship is the danger of losing your significant other due to death.

Don't get me wrong -- there's nothing wrong with good relationships.  I believe that for many (if not most) people, being in a good relationship is better than being alone.  But I think that many of us think they are in a great relationship because they feel good in it, while it's nothing of the sort.  Feeling good is contextual (and often prone to a confirmation bias) so we shouldn't draw many conclusions from that.

I do think we should all strive to find the great relationship in our lives, but the truth be told, they are very rare.  Once we do, we can consider ourselves very lucky as we keep our significant other close to us.  The good thing is that since this is a great relationship, we don't have to put much effort in it – it just happens naturally.  It's a little bit like playing an instrument -- the greatest virtuoso who must put a lot of effort in rendering the music perfectly well with an astonishing accuracy still fades in comparison with a musician who can feel the music.

Where do all these t-shirts go?

In order to gain an edge over competition at Superbowl finals, t-shirt companies make t-shirts with both winning teams on them.  When the winner is announced, the company ends up selling the half of their stock that had the right team.  I've thought of this as a great example of good pricing analysis (the value of being the first one out with a product outweighs the cost of all these useless t-shirts) but not a particularly efficient thing to do.  However, a friend recently revealed one mystery to me: he was on a trip to Peru and saw all those people wearing the t-shirts with the other team apparently being the winners.  So that's where all these t-shirts go!

The Wonderful Twentieth Century

The Twentieth Century fascinates me.  Perhaps it's because it is the most recent century that passed (unlike the 19th century that seems so far away, and the 21st century that we're living in), but it seems truly unique to me.

Just think about some of the events and inventions that the Twentieth Century witnessed: two massive wars (large enough to be called "World" wars), two totalitarian regimes that enjoyed popular support for many years that shaped relations between states of the world for decades to come, the abolition of a model of the Universe that seemed so accurate, the digital era that massively increased human productivity. We began the twenty-first century equipped with computers that can perform computations a trillion times faster than we can, that control airplanes and nuclear power plants (none of which we had in the 19th century). We were are the brink of destruction (at our own hands). We sent a man on the Moon and a machine on the Mars.

These are incredible achievemets (whether positive or negative).  I'm very curious (albeit not without a sense of nervousness) to see what the twenty-first century will bring (the very first year of the twenty-first century will, unfortunately, be forever remembered because of the tragedy that swept the U.S. and the world).  Will we have tapped into unlimited, renewable energy sources?  Will we start living away from the Earth? Will the governments of the world turn into one or will Google run the world? Will we witness Singularity? Will we destroy all life on Earth?

Sometimes I think that the future won't be as momentous as the twentieth century was - after all, it's been eight years already and I don't see anything worthy of mention happen. Perhaps we, used to the exponential growth of technology on one hand, and humbled by human tragedies of the past hundred years on the other, feel we can't be surprised again (one of my worst fears when I was younger was to grow up and be out of touch with the cutting edge- you know, the way you cringe when your parents don't know how to use a remote control. Lately I've been revising my thougts on this- my generation, unlike that of my parents, has learned to adapt to rapid change and progress, almost taking it for granted, and I think this will ensure that I'm on top of what's happening and perhaps be able to beat my son at his favorite video game).

Mankind's Only Chance of Survival

I am convinced that mankind's only chance of survival is the discovery of an efficient, renewable source of energy. The rationale is simple: relying on fossil fuels means that sooner or later we will run out of energy. Ever since the Industrial Age, humans started relying on energy consumption as a need nearly as fundamental as the need to consume food. Energy translates to money (if you've been to Dubai, you will agree) and, indirectly, to power. And, most importantly, the shortage of energy, just as the shortage of any resource, leads to competiton for what's left, and ultimately wars.

It seems to me that assuming the current population levels, the energy scarcity will begin to threaten mankind's progress, and, ultimately, existence (energy is the most fundamental resource- if we run out of it, it's game over. If we have plenty of it, we can probably ensure we'll have enough of other resources, such as food).

What we should be doing now to avoid extinction, then, is collaborating to discover such an efficient and renewable energy source.

Why is efficiency a factor? Our progress depends on how quickly we can generate energy, and-more importantly-how effectively we can store it. Fossil fuels are a relatvely efficient energy source which can be stored quite well. Our current endeavors in the space of renewable sources fall short of this requirement-solar cells generate very little power for ther size, batteries aren't nearly as good at storing energy as, say, oil.

The Sun provides us with-for all intents and purposes-unlimited energy.  The way I see it, we have two options: get better at taking the Sun's energy, or mimic its process. Scavenging it efficiently, and storing it, continues to be a challenge, but we have made some progress on fusion.

I'm optimistic and believe that we'll get smarter faster than we'll use up our current energy sources. But one should not get complacent- the future looks very dim if we fail.

The World at our Feet

As we climb the ladder of education, and especially in college, we think that the world is at our feet.  When we graduate and get a job, we forget about these aspirations and instead start thinking very small (for example, replacing the goal of fighting world hunger with the goal of getting a car).   And while in college we are naïve and overreach, later in life we underachieve.

Why we do that is still unclear to me; I think it has to do with a sense of complacency that being a grown-up brings with it; or us being hyperpragmatic since by the nature of most jobs they don't require us to think so we use our "mind cache" all the time.

Let's think big.  The world is still at our feet.  As my mentor said, we need to shift our thinking from "I want to do everything, but I can't do anything" to "I can do anything, but not everything I want".  For most of our desires are reasonable and at least some are achievable within our lifetimes.  We just need to start thinking.

What is meta and what is not meta

There was a period in my life that was clearly marked by my exuberant fascination with everything meta.  Now I treat it as a remarkable but not indispensable concept.  I remembered it after a post that contained some (really unjustified) hate for all things meta.

The concept is too simple to make an entire blog themed after it, I think, so the blog died but I'd like to cite one post I submitted there that explains my view on various ideas in the space and disambiguates them from meta proper:

meta

One can think of meta as a function that takes an object and returns an object that describes the input object using the technique characteristic of the object itself. MP3 metadata, for example, is data about the mp3 file, but not the audio itself (for example, artist information, album cover picture, etc.). It's meta because, just like the mp3 file is described by data (sets of bits), the metadata is described by the same means.

In real-life meta is shorthand for a situation in which instead of the object, the focus is on the description of the object that shares the same characteristics as the object itself. For example, suppose we are documenting the goals of our projects. One project's goal could be to understand the relationships between customers' preferences and their purchasing power. Another project's goal could be to determine all players in the currency exchange market. If we step back, we might want to determine what's the goal of us documenting all those goals. This is a meta goal. It doesn't refer to the object itself (documenting the goals) but it describes it, and uses the same technique (determining the goal).

It is crucial to understand that sharing the characteristic of the underlying object is a necessary condition. In the example above, for example, we could do many things with the given collection of goals to describe it -- we could determine their number, for example, or figure out that they all take just one sentence. But it's the fact that we're determining the goal of writing up those goals that makes the situation meta.

Not everything can be metafied, but I claim that once it's been metafied, it can then be metafied ad infinitum.

self-reference

This is most often confused with meta. If something refers to an instance of itself (either a different instance or the same), then it's self-referential. For example, if you're playing an RPG game and your character instantiates an RPG game, then you're using self-reference. Self-reference is often used to provide paradoxes: for example, take the following statement: "There are two misteaks in this sentence." There is only one mistake in this sentence, but the fact that the sentence failed to provide an accurate count of the number of mistakes is a mistake in itself -- so there are two mistakes in the sentence. Which means that the sentence only contains one mistake. But then... (ad inf.) The sentence is self-referential. Self-referential statements can cause infinite loops (but iterative, not recursive infinite loops), like in the example above.

recursion

Recursion is a special case of self-reference when the object invokes a new instance of itself rather than refer to itself. The sentence in the above case is not recursive because it does not invoke a new instance. Similarly, a piece of code that does recursion is itself not recursive: it's, however, still self-referential. An example of a recursive statement is "n factorial is n times larger than n-1 factorial". This is recursive: to determine what 5 factorial is, you have to instantiate this sentence multiple times. In the mistakes example above, you don't need to instantiate multiple copies of the sentence.

reflection

Reflection is the ability to inspect one's own definition. Reflection is a special case of meta.

reflexivity

An operation is reflexive if it operates on itself. This is not really self-referential (is the statement "my dog is cleaning himself" self-referential?).

self-relativity

It seems to be that self-relativity is the ability to relate to oneself. A simplest example of self-relativity is the word "I". If something is self-relative, it is capable of producing self-referential statements.

deep

A lot of things in the world are just deep. They're not meta or self-referential. We like talking about them because they seem complex. That doesn't make them any less sophisticated or interesting, but a distinction must be made. An example of something that's not really meta or self-referential is the barber paradox:

"There's a barber in the town that shaves everyone who doesn't shave himself. Who shaves the barber?". This is paradoxical because if the barber shaves himself, then by the first statement, the barber cannot does not shave himself. If the barber doesn't shave himself, then by the first statement he shaves himself.

Actually, this is not really a paradox -- there is a flaw in how the first statement is constructed. Let S(i) be a function that assigns to every person i a person that shaves them. Then S(i)=b ^ S(i)=i encodes the first statement. It is meant to be a tautology (since the statement describes something about this hypothetical world that's supposed to be true for everyone in the town). But if we substitute b for i, we get S(b)=b ^ S(b)=b which is false. Hence the first statement cannot be a description of the world. In other words, it is impossible for the barber to shave everyone who doesn't shave himself. It's a little like saying, "The square root of every integer is an integer. What is the square root of 2?" and claiming it's a paradox. In actuality, the first statement is simply false.

In any case, this is not really self-referential; it's just deep. The fact that the barber shaves himself isn't self-referential, but "shaving" becomes reflexive when it's applied to the barber.

What is Evolution?

Evolution has fascinated me ever since I learned about it.  I spent some time thinking about it, what it really is, what its purpose is, what its limitations are.  As many people, initially I anthropomorphized evolution, almost making it into something by itself intelligent, purposeful.  I quickly realized that evolution is nothing else than a statistical process related to survival bias -- it's a manifestation of the fact that over time, as species compete for resources, the random mutations that happen to improve the odds of one species surviving will become dominant.  We, the outside observers, will see this as the betterment of species, but evolution is nothing more than an epiphenomenon.

Evolution becomes interesting when we consider the role humans play in it. People believe that the shape of our hands enabled us to create and use tools which had a tremendous evolutionary advantage (these tools were a de facto extension of our bodies; they could be produced, used and discarded well within the lifetime of an individual).  This ability led to an emergence of intelligence, which ultimately obviated the need for mutations: instead of waiting for a random mutation to take place over many generations to adapt to changing conditions, humans can adapt much faster through being smart.  If a new Ice Age hit the planet, instead of waiting for mutations to occur that would offer some protection from the cold (such as thick fur), humans can simply leverage their ability to use tools to build dwellings that protect them from the cold, or skin animals to wear their fur.

In a way, in an age of intelligent creatures, evolution is obsolete -- i.e. humans will no longer need to evolve since the mutations no longer carry significant comparative advantage.  At a risk of making evolution into something it's not again, one can say that evolution made itself obsolete, since the emergence of intelligence happened through an evolutionary process (whether intelligence is a necessary step in the evolution of species is an entirely separate and fascinating question; I'll probably come back to it at some point).

As the epiphenomenon of evolution has been replaced by the phenomenon of intelligence, the prerogative of survival has been replaced by the mission of betterment.  This "betterment" is simply an extension of the work of our tool-building ancestors, who perceived problems and use the tools to solve them.  The creation of more elaborate and abstract tools allowed humans to solve more and more fundamental problems that they face, an, in an extreme, also prevent future problems (so, in my view, string theory is no different than a sharpened stick that our ancestors used to kill animals).  And as intelligent creatures attempt to achieve progress at an ever-increasingly faster rate, they begin to reach for more and more fundamental devices, eventually imitating nature itself.  The next step for intelligent creates, in my view, will be the mastery of genetic engineering -- the human body being the kind of ultimate tool at his disposal (at which point mutations will be replaced by direct changes to the human DNA in order to fulfill specific goals).  Eventually humans will breach the final limitation, that of the biological medium.  In that final breakthrough, what is natural will be replaced by what is conceivable (a very primitive example of this is artificial intelligence).

As we see, more primitive (and constrained) processes are replaced by more advanced ones (either those that take less time to effect a change, or those that break down more limitations).  In a way, then, we're dealing with an epiphenomenon of meta-evolution, the "survival" of large-scale processes observed in species.

It's easy to misunderstand evolution.  My mentor said that evolution as the natural move toward better should be understood as the greatest single force in the universe.  He also said that outcomes consistent with natural laws ("good" outcomes) will likely lead to rewards and this is why we should all be striving for achieving outcomes contributing to evolution (as the latter is the fundamental natural law).  I think it's a somewhat superficial understanding of evolution as something special -- an instance of the aforementioned anthropomorphizing of evolution.  There is nothing special about evolution other than it being an example of how some processes in the Universe that tend to increase entropy can exhibit patterns consistent with evolution.

Thinking About the Right Things

Part I

Many thoughts race through my mind.  I fact, I wish I had more time to think -- in my life, and particularly at work, I often default to my intuition (which, together with common sense, I see as a kind of cache for the thought process) rather than taking the time to step back and think.  It's a pity because (a) I don't think my intuition is particularly strong and (b) I like thinking.

Recently I started making myself think (and use this blog to record these thoughts and hopefully have other people provide their feedback on them).  As I think, my mind naturally collects these thoughts into themes and climbs up the abstraction ladder.  Very quickly I get to the metaphysical, the highly abstract concepts and then I get lost and confused.  I start doubting everything and asking myself existential questions.  This is why I really want to come up with some kind of framework to help me think about the right things, so I don't feel this anxiety.

I mentioned already that I think the most important thing to think about is one's purpose in life.  I still need to prove it (to myself), though, and hopefully come up with a way to do this well.  And perhaps purpose is not even the most important thing to think about, perhaps there is something even more fundamental.

The search for the most fundamental question is on.

The first stage of this search will be for me to sieve through the chaos of my thoughts and extract bits of information which might be useful in my thinking.  For example, I intuitively feel that there are some principles I want to abide by.  Maybe these are the wrong principles, but it's a good start.  Once I have all this stuff written down, I'll stare at the piece of paper and then actually think about the framework.  It's just really hard to do with a blank piece of paper in front of me.

Part II

I think it's much easier to think about the framework you should apply when thinking about your goals and purpose than to actually come up with the purpose.  I'll attempt to provide the framework I came up with.  A nice thing about the framework is that it should be independent of one's values, goals, and purpose -- there may still be multiple different frameworks but they should be equivalent; modulo perhaps some things they are trying to optimize for (some framework may try to help you come up with a design more efficiently than some other, for example).

At the most fundamental level, this framework will help us answer a very simple question: "What should I be doing right now?".  We all ask ourselves this question many times (or wish to, but don't): "Should I be working at my current job or not?", "Should I buy a house?", "Perhaps I should be helping poor children in third-world countries...".  We make some decisions but often we're not sure whether our reasoning for our choices is correct.  Ultimately, then, this will be the framework to guide us in making decisions in life.

Let's start with goals because I think a lot of people viscerally understand what goals are.  Goals are not the most concrete item but also not the most abstract (i.e. they influence other things in our life (for example, our actions), but are also shaped by other things), so we'll have to explore goals in two directions -- down (to the more concrete) and up (to the more abstract).

You may have been taught that goals should be SMART -- specific, measurable, attainable, realistic, and timely.  I'd actually simplify this mnemonic a little and say that a goal needs just one thing: to be measurable (i.e. you have to know when you've achieved it).  For now I'll assume that you already have a goal in mind that is measurable.  Most likely it doesn't have a timeline attached to it, and you're not even sure if you can achieve it (other than a visceral feeling you have).  This is fine, we'll handle all these problems shortly.

The next step is to come up with a way to achieve that goal, a design.  This is where art comes in.  There is no formula for coming up with great designs.  There are some properties that great designs have, however, and your design should also exhibit these properties:

• A good design is not sensitive to small changes in the inputs: if you miss some deadline, or if you catch a cold, a good design should be able to withstand it
• A good design anticipates any problems you may run into when trying to satisfy your goal
• A good design can be iterated upon so that it can be fine-tuned and altered
• A good design is modular -- it consists of a list of subgoals that connect logically

You should be able to prove that your design satisfies your goal.  This is usually hard, which is why having a modular design is a good idea -- you should be able to prove at a high level why achieving subgoals X, Y and Z will achieve your goal G.  We've just replaced our goal with three subgoals and a thin layer of "proof" connecting them all.  Each subgoal needs a design, so yes, it may seem like there is much more work to be done, but these subgoals X, Y and Z are now most likely easier to measure (perhaps instead of a binary -- "Have I achieved this goal?" -- you can come up with a continuous -- "How much of this goal have I achieved?" -- measure).  They may also be more timely (it's good to get early feedback and if all you have is a binary goal and five years to achieve it, you are risking a big waste of time!) and specific, and, most importantly, they feel much more attainable (because they require less work to fulfill).  And so as you come up with your <em>tree</em> of goals, the lower you get, the more SMART each goal gets.  Ultimately, then, to fulfill your goal you just need to fulfill each of the "leaf" subgoals (and make sure that your logic for connecting subgoals is correct at each level).  If at any point you are unable to come up with a design, you have an unrealistic or unattainable goal and you should alter your design.

Now that you have a design you can start executing on it.  But you won't get perfectly lucky -- problems will arise that appear as outcomes in conflict with the outcomes you expected.  A very smart man Ray (usually I fake out the initials but I hope he won't mind me unanonymizing him) taught me a way to use these outcomes to refine the design (in fact, I owe my inspiration for coming up with my framework to him).  The idea is simple: observe the outcomes, compare them to the outcomes you expect based on your design, and if the outcomes are in conflict, you have a problem.  You can work around the problem, or solve it superficially, but the most effective thing to do is to alter your design based on fundamental, root causes of the problem.  For example, if you're a poor driver and keep smashing into walls, the solution is not to buy you a stronger car; it's to improve your driving skills.  If you alter your design according to the superficial causes, you are likely to encounter more problems that are due to the same fundamental cause.

In fact, in a robust design, you can anticipate all problems, diagnose them and change your design even before you started executing on it!

As you achieve your subgoals, you should make sure that your higher-level design is still correct by re-evaluating whether achieving your subgoals leads to a fulfillment of the overarching goal.  It's possible that your "proof" was weak and you didn't take into account some inputs that have changed since you originally came up with your design.  Or it was insufficient and you actually need to fulfill more than the subgoals you originally came up with.

Hence, assuming you know your goals already, you should be able to successfully execute upon them with a help of a simple design.

Now, to the much more interesting question: how do you come up with your goals?

Part III

Our journey is not yet complete -- since we still don't know what our goals should be.  So let's travel upwards, determining what influences our goals, then what in turn influences that thing, and so on.  Hopefully the framework is a finite one.

Our goals are influenced by our desires, our wants.  At some level there isn't much difference between goals and desires, except that desires may not necessarily be measurable.  Desires are what makes our goals important: the goals then become a way to hold ourselves accountable for getting what we want.

This hasn't made our search much easier, because we can still not know what our desires should be.  At this point a lot of us have an intuitive understanding of our desires ("I want to be rich", "I want to look good", "I want to help starving children in Africa", etc.) but I doubt many of us ask ourselves why this particular desire and not the other.  This is particularly interesting as throughout our life our desires change.

I will single out one desire that a lot of people seem to have: happiness.  I don't like this word because it's not very well-defined and, even worse, as one attempts to define it, one realizes that it's self-referential (we want to be happy but ultimately we and only we decide if we're happy).  I will write more about happiness but I think fundamentally, happiness is linked very much with a concept of an afterlife (people who don't believe in an afterlife want happiness in their life).  If you think that you desire happiness, I'd encourage you to think why and what exactly you mean by this.  Would you rather be content for all your life or unhappy for most of it and incredibly happy at the very end?  Would you prefer to know more but be less happy, or be ignorant and happy?  As you start asking yourself hard questions, I think you'll realize that happiness is a kind of black hole that may actually be distracting from that ultimate question.

Our desires are influenced by our values.  Everyone has a sense of values and based on these values we make decisions in life, specifically what we think is important, and hence what we want.  Here some of us might value mankind, others may value freedom, others still may value human life (as in, an individual's life).

Finally, our values are set by our purpose.  I see the purpose as the singular value that we hold most dear, find most important.  From the purpose we can derive all our values.  Our purpose is usually also something that doesn't change -- once we figure it out in the first place, that is.  This is also another reason why figuring out your purpose is incredibly important -- the entire framework is a kind of a chaotic system where small changes to one layer can snowball into huge differences in the layers below.  For example, if our values shift a little, our goals may be entirely different and so our actions will be nothing like what they were before (often they will be opposites) and they will permeate our lives.  For example, if I start valuing quality relationships over their diversity, I may decide to stop commuting from the City and live in Connecticut instead.

Can we keep going?  What influences our purpose?  I think that we come up with our purpose based on our synthesis of our interpretation of the observable Universe.  In other words, we absorb information through observation, analyze it, and find themes.  As we continue synthesizing, we determine a very high-level model of the universe and with it, our place in it.

The reason this is a useful framework is that it is not infinite -- you can't ask "why" forever, because information is a fundamental phenomenon -- we observe things about the universe, which we can treat as exogenous.

However, this doesn't make thinking about purpose easy.  You still have to take information in, and process it; find themes and reduce it to a model that you can reason about.  You will likely encounter the metaphysical layer pretty quickly.  While you may not be able to reason within that layer, you will probably be able to do some metareasoning.

Specifically, as you start asking yourself "what's the point of all this that I see around me?", you will have to answer an inconvenient question.  I think I found a way to phrase it that gets to the bottom of a lot of the questions people have about the Universe.  Do you believe in an afterlife which can be materially affected by the things you do in this life? (note that you may believe in an afterlife and still answer "No" to the above question!)

If you don't, your purpose in life will naturally be larger than you -- it will be about affecting the universe in a way that's not simply localized to your life.  For example, I believe, your purpose in life should not be to have a house, a well-paid job, and a family -- these things affect mostly you and nothing else.

Playing with Color (part II)

(Originally published on June 20, 2009)

Last time we took an image that displayed a message which would disappear when the image was rendered in grayscale. A much more interesting problem is the opposite one — can we embed a message an in image so that it appears when the image is rendered in grayscale?

It’s not as easy as the first problem — there we had information theory on our side (the conversion to grayscale destroys information as three components — red, green and blue — get converted to just one — intensity). Now we have to get a little more creative (again, you should be able to improve on my proposed algorithm).

Instead of using information theory, we’re going to use psychology: we’re going to trick the brain.

The idea is pretty simple, but it requires us to think in the HSL space: instead of thinking of color as a composite of three primary colors, think of a color as a composite of three properties: its hue (“where on the rainbow the color is”), its saturation (“how vivid the color is”) and its lightness (“how close is the color to white”). Lightness is not quite the same as intensity (because hue and saturation are nonlinear in the red, green and blue components). It turns out that our brain decomposes the colors into their HSL components better than their RGB components (which one of the two colors has a higher R component, black or green?). The idea is to encode the message with a color of a lower intensity that compensates for it by having an unusually high lightness.

First, we’ll define a scratchpad — a two-dimensional set of all possible colors of a given intensity I (I used I=0.3 experimentally). The color space is three-dimensional, but if we add one constraint (intensity has to be equal to I) we can map the resulting colors onto a two-dimensional space. We will use the x-coordinate to determine the intensity of the green component and the y-coordinate — of the red component. We would like the pixel color to be uniquely identified by these two values. This is often possible, but if the green and red intensities are outside of certain bounds such a color won’t be possible (for example, consider a pixel whose green and red components are both 0.5 — if you want the pixel to have an intensity of 0.3, the blue component would have to be equal to -1.25).

Hence, we’ll transform the x and y coordinates so that the resulting image is rectangular — that is, for each row, x=0 will map to the smallest value of the green component that makes a color possible, and x=max will map to the largest value of the green components that makes a color possible.

If we didn’t transform the coordinates, the resulting scratchpad would look like this — note the areas that make it impossible to construct a color:

All possible colors of intensity 0.3

Now transform the coordinates (this gets rid of the empty space and expands the rest). Click to cycle through the scratchpad, its lightness map, and its intensity map (the latter should be all homogenous color, equal to I)

An expanded version of the above set of colors

We’ll take advantage of two properties of such a scratchpad: every pixel has the same intensity; and pixels that are close are of similar colors (since the mapping is continuous in the red — as we move up and down the scratchpad — green — as we move left to right — and blue space — because blue is derived from the intensity formula, which is linear).

Now go through the scratchpad and find pixels that have unusually high lightness values and make them darker. By changing the lightness we’ll be changing the intensity of the pixel but because these pixels have high lightness our brain won’t be able to determine the difference in intensities. I use a scaling function with a gamma correction:

if(lightness>I)
  lightness = ( ( (lightness-I)/0.35 )^0.8 )*0.2 + I
end

Click on the image to cycle through the scratchpad, its lightness map and its intensity map (this time the latter should have a darker area):

We've altered some pixels to lower their lightness

Now, if you look at the resulting image, you will see that a number of pixels have an intensity different from I but they still look pretty bright (because their lightness is high). We’ll pick either the dark or the bright pixels based on whether we’re encoding the message or the background. We limit the colors we pick to a subset of the scratchpad that features pixel ranges with the highest changes in intensities (that way we can “smuggle” a low-intensity pixel in around similar pixels of a higher intensity).

Secret message embedded using pixels of lower intensity but unusually high lightness

We can stop here — transform the input image the same way we did it last time, by decreasing the variance between white and black and pick a color from the scratchpad that matches the intensity. But this is not good enough (our brain is, after all, smart). Let’s introduce a distraction message that appears faintly in the color version of the image but disappears in the grayscale version (we already know how to do that!). Based on the intensity of the respective pixel in the distraction image we’ll alter the color we picked from the scratchpad without altering its intensity (so as to cause no difference in the grayscale image).

Added a distraction image

We’re done — we’ve successfully tricked our brain (provided that we pick a smart distraction message).

Note that on your computer, you will still be able to see the message in the image above, for two reasons.

Depending on which browser (!) you’re running, your browser may embed color profiles in images to compensate for differences that various computer screens render colors. This, of course, screws our mathematics up somewhat.

• If you’re viewing this on an LCD screen (and, let’s face it, you are), you will be able to change the angle from which you look at the screen to reveal the hidden message.
• If we print the image out, however, we’ll see the full effect. To fix it so that the message disappears on a computer screen as well, we can simply apply the inverse color profile transformation, or define the region of the scratchpad that we pick pixels from differently.

Demo!

You can see a demo of this effect here

See also

Check out References for some useful color-related references, including a note on color space conversions.

Check the code out on github.

Playing with Color (part I)

(Originally published on June 20, 2009)

I’m fascinated by color. When I was younger I dreamt of discovering a color that nobody has ever seen before. And since most computer languages give you easy ways to manipulate graphics, I played with color on a computer.

One day I saw an interesting effect on some certificate I received. When I photocopied it in black and white, a word appeared in the photocopy ("COPY") that wasn’t visible in the original. This "security measure" took advantage of the photocopier’s inability to faithfully replicate the document (the word COPY was actually visible in the original but it was difficult to see because it consisted of a myriad of tiny dots). I wondered whether it would be possible to hide messages in documents even if the copier were to produce faithful facsimiles, but converted color to black & white images.

It turns out that it’s possible, and not that difficult. I’ll explain how, while trying to limit distracting you with color theory to the minimum.

On a computer screen, each color is a mixture of three primary colors, red, green and blue (printers use a slightly different scheme but for all intents and purposes we can limit ourselves to talking about the RGB space). It turns out that pure red does not appear as “bright” to the eye as pure green. In fact, the intensity of a color (how “bright” it appears to the eye regardless of its hue) has been found to be a weighed average of the three primary components:

intensity = 0.299*red + 0.587*green + 0.114*blue

This intensity is precisely what you see when you convert an image to grayscale (note that you may find different formulas depending on what devices you’re dealing with — different devices have different sensitivity to various primary colors). For example, consider the following image and its grayscale version:

Not all colors were created equally bright

A simple idea for hiding messages is therefore to produce an image of pixels whose intensity is all the same, but whose hue is different (so that we can see the message in color). For example, we can give every pixel of the message the color of (1, 0.3, 0.4) and every pixel of the background the color of (0.3, 0.7, 0.176). The intensity of the former is 0.52, as is the intensity of the latter. Since the intensities are close to each other, when we desaturate the image, the message will blend with the background.

This works well if we know the relationship for a given device exactly. Some printers (if we wish to print the image out) use a slightly different relationship (or perhaps they round the weights). In such a case the two intensities will not be exactly the same and the eye is surprisingly good at telling the difference between two close colors if they cover large areas! We can improve this algorithm by adding some randomness (this is just a suggestion that works pretty well, you can find much better algorithms!):

Consider an input image that contains the message (white text on black background). Each pixel has an intensity of 0 (if the pixel belongs to the background) or 1 (if the pixel belongs to the text), and probably somewhere in between if the text is antialiased.

The secret message we'll try to hide

First, transform the image by adding some random noise and reducing the variance between white and black:

targetIntensity = (rand() + sourceIntensity + 1)*0.33

Adding some randomness

Once we have the target intensity, we’ll randomly adjust the intensity formula and find an output color such that the intensity formula cancels targetIntensity out:

alpha = 0.587 + (rand(25)-12)*0.001
beta = 0.299 + (rand(25)-12)*0.001
gamma = 1.0 - alpha - beta

ratio = (alpha-gamma)/beta
r = 0.44 + targetIntensity/3.0
g = 0.62 - targetIntensity*ratio/3.0
b = 0.9 - targetIntensity/3.0

Convince yourself that the intensity is a constant, independent of targetIntensity. But since targetIntensity varies, the color version of the image will consist of distinctly different pixels.

As a nice touch we can alternate between two different formulas for added noise:

The resulting message that disappears when rendered in grayscale

Print the above image out in black and white and see the message disappear!

Demo!

You can see a demo of this effect here

See also

Check out the second part of this post (hiding in the color rendering) here.

Check out References for some useful color-related references, including a note on color space conversions.

Check the code out on github.

I wish humans had 12 fingers

We count in base 10 because we have 10 fingers.  10 is not a particularly good number to use every day: it divides only into 2 and 5, so many everyday operations (such as splitting something into a small number of equal parts) are just painful with a decimal numbering system.  Of the remaining divisors – 3, 4, 6, 7, 8 and 9 – all but two yield nasty infinite decimal expansions (because 3, 6, 7 and 9 factor into primes that don't appear in the prime factorization of 10), 3 being the most painful one since it appears in nature so much.

I wish we had 12 fingers.  12 is a number that is far superior to the number 10.  It divides evenly into 2, 3, 4 and 6 parts.  Dividing into 5, 7, 10 and 11 still produces infinite expansions, but who needs to divide into 5 in nature anyway.

A friend of mine noticed that if the additional two fingers were opposing, like our thumbs, our hands would be infinitely more agile.  Just think about the sorts of manipulation (and hence the sorts of tools) we would be capable of with two thumbs at each hand!

An iterative approach

I struggled with publishing my blog for a long time for two reasons.  For one, I guess I didn't really internalize its importance (which arises from a connection this blog has to my goals).  As I'm solidifying my goals and training myself to stick to them, I post more.

The second reason, a much more subtle one, was that every time I started thinking about what to post, I'd get discouraged because what I was going to talk about wasn't fleshed out, or fully thought out; in other words, it wasn't perfect.  Perfection is a very dangerous concept -- I think we all seek it yet we don't realize how unrealistic and unachievable it is.  We're drawn to it because of its elegance -- you never have to fix a perfect thing -- and forget an obvious thing that the closer you come to perfection, the more effort you have to put in (and the relationship is exponential).

Instead, I should much rather aim for something that gets me in the vicinity of the right answer and then iterate as necessary, by contrasting the cost of improvement against its benefit.

Specifically for the blog posts, worrying about making the posts perfect meant I'd never post anything.  Instead, I'm just going to try to convey the gist of my thoughts and if need be, elaborate on them in the future.

Good principles to stand by

Collaboration. I think that collaboration and sharing are important things. I want to share a lot of things I came up with with everyone. I feel collaboration may be one of the principles relevant to my purpose on life.

Iteration. Instead of attempting to get something perfect, I'd prefer to iterate over it.  Note that iteration is not equivalent to incrementalism – I'm not claiming that every iteration will necessarily add a certain amount of value. Such a thing would imply a greedy approach and not every problem can be solved with a greedy approach.  Instead, I want to come up with something that adds value given the information I have now and given an appropriate (not large) amount of effort, and then decide whether to gather more information, put more effort, or do nothing depending on the cost and the benefits of either option.

Logic. I think we can derive a lot in our lives simply by using logic.  A logic-based approach also forces us to clearly separate our process (the rules by which we come up with conclusions) and our axioms (the fundamental concepts we believe in, which we use to build our world view, and which we can't derive from other conclusions we've come up with).

Respect for mankind. This one is tricky.  I'll need to elaborate on it soon.  In the meantime, we have to define "mankind" (we can do this by identifying what's unique to mankind... is "the sense of purpose" unique to mankind?), "respect" and the reason why it's an important thing to respect it.

Efficiency. It's important not to confuse efficiency with being in a hurry: a lot of people rush to do things (I do as well, incidentally), but still do them inefficiently (the difference is not how much output you produce, but how much output you produce relative to your input). I get excited when I manage to plan things out so that they all fall in lockstep; one thing happens after another, like the mechanism that makes a clock tick (or a less linear version of a Rube Goldberg machine).

The most important thing we should all be doing right now

The most important thing each one of us should be doing now is figuring out our purpose in life.  Some of us have already done it; I envy you.  It's difficult and time-consuming, but not as difficult and time-consuming as one may think, which is what I realized a couple of weeks ago.

Why is it so important to know your purpose?  Because purpose allows you to define your values, which allows you to define your goals in life, both the distant, momentous ones, and the small, short-term ones.  Once you know your goals you can come up with a plan for what to do, and finally do it.  So purpose, in my view, flows all the way down to everything we do.  If you ever tried to assess your actions, wondered what the right thing to do is, hesitated before making a decision, you have indirectly tried to figure out your purpose (only at a much lower level).

At first finding purpose seems like a daunting task because it forces us to introspect, to think carefully about our wants, to define our philosophy, to face our fears.  It seems so removed from our daily life that we underestimate its importance and choose not to be bothered with such abstract ideas as a purpose.  Most of us have probably never asked ourselves this question.

While there's no right or wrong answer, I believe each one of us can find <em>an</em> answer that will be consistent with our intuition and deeply satisfying.  I haven't found my purpose yet but I imagine that I'll be much happier once I do; I'll be able to motivate myself more easily, be more confident, and feel a sense of pride.  Most importantly, that annoying little voice in my head that keeps asking "Are you sure you're doing the right thing?  Are you sure you're not wasting your life away?" will go away.

There is no right or wrong answer but there is a right and a wrong approach to finding it.  You can't do it arbitrarily, by guessing or reading it in some book.  You can't ask your parents, your spouse, or your peers.  The answer has to come from you alone.

I'll be harping on about this more in the future because I think it's unbelievably important.  For now, I'll say that I started thinking about my purpose, and I believe that I will figure it out.

What is elevenseconds?

Perhaps it's...

• the amount of time it takes me to fall asleep.
• nothing. I'm just messing with your head.
• an homage to a hockey accident.
• the amount of time it takes us to boot up, right after we're born. It's longer than we'd expect; for some reason God chose hibernation over standby.

Reflections of a 24-year old

Let me tell you a little bit about myself.  I have not once in my life stepped back to reflect on my life, values, and relationships.  Until tonight.  It's strange that it took me 24 years to figure it out but the mind, which one usually employs to do hard tasks whose execution is in some way related to your paycheck, does crazy things when one lets it just roam freely.

24 years old.  I don't feel old.  I feel creative, full of potential (some of it untapped even by the Man himself), I feel refreshed.  I finally "got it".  The little scraps have been flying through my mind since forever, although in the past few months they have intensified.  Now it all makes sense -- and while it would be a cliche to try to encapsulate it in two words, I'd give it a try:  Live free.

Live.  At the end of the day, that's all you have.  Your life.  It's your own little movie, with many actors (one of them you), a script (written by you), a director (you), even the stunts (yup, it's all you).  There is nothing in this world you can take better control of than your life.  Many people forget about it -- they allow their life to control them; they are defined by the routines they do; they convert the simple active "Live" into a passive "Let your life by lived by you".  This seems awkward, doesn't it (it's grammatically correct, of course).  But more than that, it literally suggests that you should allow this ephemeral, esoteric thing ("life") take the lead ("Let your life", as in "Allow your life"), putting "you" conveniently at the very end of the sentence.  Don't do that.  Live.  Active.  Don't be a spectator in your little movie.  Be the cast!  Don't let your life be controlled by the collective, the "Civilization" (more about that later); don't be a puppet whose strings are pulled by the shared conscious of mankind.  Be an individual (a brilliant scene from Life of Brian comes to mind).

You heard it right -- "Live" is an imperative.  It's something you're supposed to do (you may not think of it this way, but the way I see it, the originator of this imperative is "life" itself, which adds the whole thing its devilish meta twist and I love meta); it's a rule but shouldn't be seen as one -- rather, it should be seen as a subordinate phrase in a conditional -- sorry for my linguistic snobbism -- "If you want to [X], Live freely", where [X] is pick-your-poison.  It's whatever dominates your utility function (so by definition, the premise is true: you DO want to [X]).  This framework is also helpful because it shows you a very important principle -- that the best things in life don't come pre-packaged -- whatever wisdom you get, you have to synthesize it yourself, draw the right conclusions, flavor it with the context of your life.  So "Live" is not equivalent to hedonism.  It's also not equivalent to total altruism.  It demands for you to define your utility function -- "what makes me happy? / what makes me feel good? / or from really first principles: what do I prefer to do over anything else (or a pareto optimal frontier thereof)?" -- and steer your life so that your utility function is maximized.

"Free" not in the meaning of "freedom" or "liberty" as we have so literally started to take those terms in the past few decades especially in the U.S., but in the meaning of "unconstrained".  Don't let obstacles slow you down.  You're racing 90mph down a curvy road (say, one of the many roads in Northern Fairfield County).  If you see an obstacle, you don't stop or swerve.  That could end up badly for you.  You go right through the obstacle.  Minor bruise for your car, and a decision that could have just saved your life.  What are those obstacles?  They take many forms: restraints we put on ourselves for no apparent reason ("I'm not going to understand this book, so I won't bother; I should just watch the movie instead"); "convention" and "norm" (which those two words, incidentally, are my "bad idea bears" for the day) which keeps us from being different, unique, from making an impact (ironically, mathematically "norm" is by definition a transformation that maps different instances of some entity into one instance, and is often defined to help us tame entities into something we know and expect (hence no impact) -- so by definition "norm" is an antonym to "uniqueness"); and finally, and most importantly, problems in our lives, the little ones, and the big ones.  Most people let those problems bury them; they are like a smog that prevents you from seeing ahead, from seeing your life and the paths it could take; that prevents you from having a vision (literally and metaphorically!) of your life; that prevents you from being great.  Don't be one of those people.  Solve your problems.

Paul Graham (yeah Kemp/Alex/Will), one of the wittier computer science writers alive, said once "I don't hate bugs [in my code].  I love bugs!  Because there I am, against the bug, and the bug knows it's just a matter of time before I kill it.  miau."  This holds true for problems -- approach them systematically (one thing that problems love is an unsystematized approach -- like not having a plan for how to solve them, or solving too many at once, or solving each problem as its own "instance" rather than seeing patterns and killing the problems even before they are born!).  List all your problems (that's the OCD part of the process).  All of them.  It might be a long list.  Don't be embarrassed -- only you and Microsoft Word (or vim if you swing that way) will know.  Then, for each problem, figure out the root cause (that's the "Sherlock Holmes" or "House, MD" part of the process--reverse engineering things).  This is where it gets tricky -- because some of the problems you thought you just listed are actually diagnoses, so you'll have to go through the list again.  It will most definitely be a long and arduous process (but then, just do a little each time -- surprisingly, not that many new problems enter our lives every day!).  Once you do that, figure out (that's the creative part, the "van Gogh") the solution.  That'll already take you further than most of us have ever been.  Then define a gameplan for those solutions (the "coach" part) -- what do you plan to do first, by when, etc. Finally, execute (the Pierrepoint part -- by the way, it looks like a good movie.  By the way to that aside, got lost in the tangents?  It's a stack; easy to get back on track).  A mistake I have been making for many years was manyfold: not listing all problems and conflating them with root causes, and not executing on the gameplan.  You need those two extremes to work particularly well -- they are your interface to the outside world.

(curiously, the settings of my mail window made it so that in the paragraph above the longer words "understand", "incidentally", "mathematically", "defined", "uniqueness") all line up in a visually-pleasing diagonal (the "high density" letters such as m, e, a, h make this diagonal apparent).  See if you can get that with your mail program -- try out some window widths.  I'm not going to tell you what my width is -- and yes Will, although you could actually compute it given the information I provided -- the average number of characters by which these five words I mention are apart -- I encourage you to experiment.  Perhaps you'll see patterns that I failed to see.  Rather than showing you this one pattern, I would like to teach you how to see patterns.

Speaking of teaching.  Calling it "important" would be a huge understatement.  I think the ability to teach defines mankind.  Whatever definition you take for mankind (not related to the species but to the phenomenon), it somehow rests on the notion of civilization (do that exercise! -- try defining "mankind" or "intelligence"). Civilization is this strange thing -- it's intangible yet it assumes the existence of so many tangible objects; you think of the "Roman civilization", you see the Colosseum, the Legions, Asterix, the lazy fat guy resting on some sort of hammock and eating grapes -- all tangibles!  It's man-made (does not occur in nature), it transcends man (a man dies, civilization stays), it's -- I'd claim -- the closest one can get to calling something "man's soul" -- the intangible shared information that's part of the collective), it's made possible because of the institutional memory, which one man passed to another by teaching.

(Aside: if only humans could all communicate through the mind -- what I saw in my mind as a simple equivalence took me an entire paragraph to write up).

I've taught some classes in college, I've taught the new hires at work, I've taught my brother how to take square roots (no calculator!), I've taught my parents that family, no matter what the short-term bumps or the long-term distance might tell you, is incredibly important in one's life, it's embedded in one's conscious. These are all instances of the same principle -- "sharing your passion to know/feel/think/do with others".  Everyone can be a teacher in this framework.  I encourage you all to share your passion.  Whatever it is.

Live free.  This sounds simple but in fact it's so generic it's not very useful (remember: pre-packaged stuff) unless you internalize it.  And this is the thing: you can read about it, you can be told this same thing over and over again, you can be forced to memorize the definitions and the corollaries of this, but you won't understand it, you won't "get it" unless you reflect on this phrase, let it assemble with your mental framework while it's fresh, and let it purge your framework of weed (Ideally everyone's mental framework should be one principle, the Master principle, from which everything else is derived; but as humans we like to create "caches" or "short-circuits" so we don't have to re-derive all the time.  The problem is that when we rely too much on the caches, the cache becomes a part of your framework in fact obfuscating the Master principle.  The great thing about the framework, unlike Will's code, is that it can be refactored (cleaned up, rearranged) really easily.  You just need to think about it; reflect.

So as you reflect on the gist of this email (again, this is meant to transfer my passion about this, not the individual facts or inane adages I may have inadvertently constructed), the phrase "Live free" will enter your framework, but as it passes through the plumbing, it (like a virus -- a "good" virus) has the ability to destroy those stale caches and simplify your framework.

As you see, this email, too, is very strange coming from me.  Perhaps tomorrow I'll be just my former self.  But perhaps, all the thoughts that races in my mind tonight made me realize the value of reflecting.  You should all do it sometime.  Your brain deserves it.

What's this mystical "framework" that I so eagerly refer to in this email?  A framework is a set of principles that's logically sounds (but not necessarily complete), possibly adorned by those "shortcuts", where for example, we substitute an example for a proof that one principle follows another (principle: humans are all capable of accepting teachings and incorporating them into their framework (sorry, self-referential again.  "Learning" would just unnecessarily litter our space with one more word that we have to define); derived principle: experiences in life can provide such teachings.  It's not too easy to derive it from the former (something like "we accept teachings" => "teachings must be facilitated by a teacher"; "the teacher must share with us more than what we contain in our framework" (otherwise it's a vacuous case) AND "we experience signals from the outside world" AND "some experiences are not part of our framework" (which is hard to prove) => "there exist experiences that are teachers so we can learn from them".  See -- that was hard.  So we substitute in an example "I touched a hot stove when I was a child and got burned: I learned not to touch hot things") [yeah I got lost in the parentheses, too].  If not an example, we substitute in a model (for example, we understand gravity through a proxy of the infamous F=mg.  But do we really understand gravity or just the mathematical implications of the model?).  While the example/model is a good shortcut from the whole proof, if used too much, it will replace the proof entirely and then, since in examples you don't need the statement of either the derived or the higher-level principle, you forget what those principles were.  You can imagine this process cluttering your framework all over -- it's going to be impossible for you to get to the highest level principle (there's always one -- semantically it's equivalent to saying "stick to the set of axioms you accept as true" -- because you can definitionally derive all other principles from it).

So this is why reflecting allows your framework to clean itself up.  Forget the examples, forget the specifics.  Forget the F=mg formula/model, forget the falling apple example.  Think in terms of high level principles: mass attracts mass.  [Sorry for all the physics, guys].

Why does your framework need to be clean?  Because if it's clean, you have direct access to all the principles (through the proofs), and thus to the highest level principle.  If you do, and if you learn to apply the principles to life, you will LIVE your life FREE.

If you've made it all the way to here... some lighter stuff:

Claim 1.  This is not a moral reasoning paper or an introduction to expository writing.  In fact my paper for the latter in college was an analysis of Dark City submitted as a web page.

Corollary.  Most of what you just read could be crap -- it could be fantasy, hallucinations, rambling.  Please approach it critically -- ask questions if you're interested, argue if you disagree, present counterexamples.  (Although hopefully by now the "good" virus I just planted in your head with this email worked its way into your framework and removed some of that old weed...)

Proposition 1.  I propose as means of assessing the amount of reflection one has done to count the occurrences of semicolons in some writing that is inspired by said reflection.

Corollary.  This write-up has 17 semicolons; oh wait, now it's 18.

Completely subjective opinion.  18 semicolon-worth -- that's a fair bit of reflection!