The Meaning of Emoji

This podcast is all about emoji. But it’s really about how innovation really comes about — through the tension between standards vs. proprietary moves; the politics of time and place; and the economics of creativity, from making to funding … Beginning with a project on Kickstarter to crowd-translate Moby Dick entirely into emoji to getting dumplings into emoji form and ending with the Library of Congress and an “emoji-con”. So joining us for this conversation are former VP of Data at Kickstarter Fred Benenson (and the ???? behind ‘Emoji Dick’) and former New York Times reporter and current Unicode emoji subcommittee member Jennifer 8. Lee (one of the ???? behind the dumpling emoji). 

So yes, this podcast is all about emoji. But it’s also about where emoji fits in the taxonomy of social communication — from emoticons to stickers — and why this matters, from making emotions machine-readable to being able to add “limbic” visual expression to our world of text. If emoji is a (very limited) language, what tradeoffs do we make for fewer degrees of freedom and greater ambiguity? How exactly does one then translate emoji (let alone translate something into emoji)? How do emoji work, both technically underneath the hood and in the (committee meeting) room where it happens? And finally, what happens as emoji becomes a means of personalized expression?

This CFI Podcast is all about emoji. We only wish it could be in emoji!

Show Notes

How emoji originated, how they’re standardized, and more [0:48]

The difference between emoji and emoticons [11:03]

Social and political considerations around which emoji to include [15:30]

Using stickers and images to express emotion [21:40], as well as Bitmoji [24:20]

The story behind creating “Emoji Dick,” a translation of “Moby Dick” using emoji [29:11]

Transcript

Sonal: Hi, everyone. Welcome to the “CFI Podcast.” I’m Sonal. Today’s episode is all about emoji. But it’s also about bigger questions and how innovations come about, from the tension between open standards and proprietary systems, to the economics of creativity. We begin with a tour of different emoji and how they came about, the politics of emoji, where emoji fit in the taxonomy of visual communication, and why this matters. And finally, we talk about the difficulties of translating emoji when it’s not really meant to be a language. Joining us for this conversation are Fred Benenson, an early employee at Kickstarter who built their data team. He’s also infamous for kickstarting a project to translate “Moby Dick” entirely into emoji. Also joining us is Jenny Lee, former New York Times reporter, who is a member of the Unicode Subcommittee on emoji and who recently led the effort to get the dumpling emoji, which is where we start the conversation.

Emoji basics

Jenny: I wasn’t a really big emoji user. In fact, the first time I ever heard of emoji was when Fred started his Kickstarter called “Emoji Dick.” And I was like, “What the fuck are emoji?”

Sonal: What is “Emoji Dick?”

Jenny: This was before they showed up on our iPhone with, like, perky little yellow faces. I was like, what? It’s, like — sounds [like] something very bizarre.

Sonal: I just started. I didn’t even actually — just to be blunt, I had a very hard time using emoji, because I didn’t quite understand how to even, frankly, use them. I don’t understand when people send it to me, if it’s not the obvious heart, you know, etc. But as I’ve been using it more, I’ve found myself, sort of, expressing myself now in, kind of, quirky ways. And I don’t know if people really get it or not, but I’m getting a kick out of it.

Jenny: But that’s the fun of the ambiguity.

Fred: I have a friend who showed an exchange between a friend of his who was dating a guy, and he would only send her emoji. And she was like, “I just can’t — I can’t handle this.” And he showed me the screenshots of their exchange and it was hilarious.

Sonal: You’re helping translate.

Fred: But yeah, and so, like, I was like, “Oh this is…”

Sonal: You’re like the Cyrano de Bergerac of, like, emoji.

Fred: Yeah. I was like, “This is what this means.”

Jenny: I can definitely see it being, like, sort of an irreconcilable difference between people in relationships.

Fred: Significant other.

Jenny: Fast forward many, many years, emoji have shown up on our iPhone. And I’m texting with my friend, Yiying Lou, who’s best known as the designer of the Twitter “fail whale.” So, we’re texting back and forth about, like, dumplings. And so I sent her a picture of the dumplings I’m making. And then she texts me back knife and fork, knife and fork, yum, yum, yum, yum, yum. And she goes, “Wait, Apple doesn’t have a dumpling emoji.” I was like, “How could that be?” I was, like, because there’s so many obscure Japanese food emojis, since emoji are from Japan. Like, you have, you know, everything ranging from ramen to curry rice, to tempura, to, like, you know, the rice thingies on a stick to even — there’s even, like  — triangle rice bottle, looks like it had a bikini wax.

Fred: There’s also the fish cake, which is the white one with the purple swirl.

Jenny: Yeah, yeah, the spiral. Totally, right?

Sonal: Oh my God.

Jenny: And I was, like, how could there not be dumplings, right? Because it’s such a universal food, right? Because there’s, like, pierogies in Poland, and momos, and gyoza, and empanadas. Like it’s just, like, a food from around the world.

Sonal: I mean, technically, samosa is a dumpling.

Jenny: Yeah, samosa, ravioli. And I was like, okay, emoji are universal, and then dumplings are universal. How could there not be a dumpling emoji? And just — in my mind, I was just, like, clearly, whatever system in place has failed.

Sonal: How do you solve a problem like the dumpling emoji?

Jenny: Yeah, and I found out that emoji are regulated by the Unicode Consortium, which is a nonprofit organization based in Mountain View, California. It now has 12 full voting members that pay $18,000 a year just to vote on issues, including, like, emoji and other kinds of, like, technical…

Sonal: Are those members in Mountain View or from around the world?

Jenny: So of those 12, 9 are U.S. multinational tech companies — Oracle, IBM, Google, Yahoo, Adobe, Facebook, Microsoft, and Symantec. Then of the other three full voting members, one is a German software company, SAP. Another is the Chinese telecom company Huawei. And the last is the government of Oman.

Sonal: That’s a really interesting crew.

Jenny: Isn’t it an interesting crew? And they have these quarterly meetings, and then I just show up. And they’re, you know, very welcoming. You know, they’re like, you know, “Thank you for coming. What brings you here? Tell us about yourself.” It felt like showing up at church — like a new church. You’re a new member. They all knew each other very well. They’re very excited that there’s, like, someone, you know, young and, like, diverse, who’s just, like, randomly showing up. And so I in that process learn how you get emoji passed, and how they’re regulated. And so, in January of 2016, we submitted a full proposal for dumplings, take-out box, chopsticks, and fortune cookies and got those all passed. So, those will be in Unicode 10, which means that — that’s announced in June of 2017. And so, they’ll actually hit your phones several months after that. I was like, wow, billions of keyboards will be impacted by this and…

Sonal: That’s amazing. Were there other proposals submitted at the time?

Fred: Oh, there are constantly proposals. There’s this whole process that people like Jenny — some of them make it through.

Jenny: It’s complicated, yeah. No, if…

Sonal: It’s a lot of work. It does introduce some good, useful bars actually for making sure quality gets through at some point.

Fred: Yeah. And to their credit, the Unicode Consortium has an amazing list of emoji criteria, where they say, “Okay, here’s what we’re looking for for emoji. It’s gotta have like, you know, kind of a unique meaning, in that it’s not covered by other stuff, but it also should have, like, you know, some ambiguity. So, it’s not just, like, literally one thing. It could be used in other contexts.”

Jenny: Also, there’s one of the more interesting rules, which is no celebrities, deities, or logos.

Fred: Whoa. The Easter Island head is kind of a violation of that one, but that’s got its own story. A couple of years ago, with a big update, the Easter Island head showed up in, like, the back of the travel section of emoji. And I was, like, what is that doing there? Who is traveling to Easter Island so often that they need to use the Easter Island emoji? And it kinda just stuck in my mind. And then I started using it in this, kind of, like, slightly culturally insensitive way to, like, reference some supernatural phenomenon that I didn’t understand, right? Like, if I was in a conversation with somebody and I was just, like, completely flummoxed, I’d just, like, send that one.

Sonal: Yeah, it’s like your version of Bermuda Triangle or something.

Fred: Yeah, yeah, I was just like, “Who knows? Stoneface.” Other people use it for, like, stoned, right? Like, there’s lots of combinations in there. The reason why it’s in there is that there’s a statue in downtown Tokyo. I think it’s a Shibuya station that is called Moai, which is a name of just, like — it’s a proper noun of that statue, which was made by an artist that was, like, a reference to [an] original Easter Island head. So, it turns out, Japanese teenagers use this waypoint to meet each other. And so, that’s how it ended up in Japanese cell phones, and that’s why it ended up in emoji. The artist used this inspiration of Easter Island. The interesting twist is that when you look at it on the iPhone, it doesn’t look anything like the statue in Tokyo. At some point, Apple was like, “We’re not gonna make it, like, this Tokyo one. We’re gonna do it [like] the original one.” Android, on the other hand, their Moai emoji looks like the Tokyo station one.

Sonal: So fascinating. I read a study — I actually included in our newsletter months ago of someone comparing how emojis look on different platforms and how it actually changes meaning, because…

Jenny: Totally.

Sonal: …you can actually think you’re sending one thing and you get something else.

Fred: That’s gonna happen in any system that has standardization. Like, you’re gonna try really hard to make sure people hue to the specification. But, you know, people do their own implementations and things change. In fact, the whole reason why emoji are in Unicode was because you would send your friend an emoji, and then their cell phone would actually just render the incorrect one. It could be so much worse. And the fact that there is a standard means that, like, you only get these, like, weird edge cases.

Jenny: There are still some interesting vestiges of, like, the different telcos between Apple and Google. One was Docomo and the other one was SOFTEL.

Fred: SOFTEL.

Jenny: SOFTEL. So, they’re basically — depending on who their partner was locally, they kind of inherited those generations of emojis. For example, on Apple, “women with bunny ears” is, like, two women dancing in kind of, like, a “let’s party” kind of way with their bunny ears. Whereas on Android, it’s just the headshot of a woman with bunny ears.

Fred: And it’s referencing this slightly misogynist part of Japanese culture of bunny woman, which is itself a reference to the Playboy bunny.

Jenny: Oh, right.

Fred: And so, they were cocktail waitresses working in nightclubs. That made its way into the Japanese set. And so when it came over to America, like, I think Apple must have been like, “Let’s make this a little more fun.”

Jenny: One of the easiest things actually to get emoji passed is showing that a vendor uses it. Another argument is for completion. This is actually why chopsticks got passed fairly easily, because we had, like, knife and fork, so you need…

Sonal: Oh, so you need completion of a set.

Jenny: …completion. So that…

Sonal: So, it’s actually you can tell a whole story, like, stringing together a bunch of…

Jenny: No, I just think that it’s, like, they’re engineers…

Sonal: Right. You can’t have ABCD and skip the D.

Jenny: Yeah, yeah. Actually, one of the weird issues is that there are red, yellow, green, purple, blue hearts…

Fred: Hearts. Yeah, yeah.

Jenny: …but not orange. So one of the big lobbying efforts has been to fill in the orange.

Sonal: So the case of the Apple bunny ears and the Japanese bunny women — that was a case where there was an intentional translation to, sort of, obscure the cultural reference.

Jenny: It’s more that…

Fred: There are just two separate ones, right.

Jenny: …they’re often — try to map technically the same emoji, but it’s, like, rendered and sort of interpreted differently. They like emoji that can have multiple meanings. You can also just have, like, emoji that have one meaning. But it really has to be a really good one if it’s gonna be one meaning. So for us, the Chinese take-out box, for example — one of the arguments that we made is that, one, it’s an iconic shape. It also symbolizes both an entire cuisine, which is Chinese food, and also a means of eating, which is delivery…

Fred: Takeout, right.

Jenny: …and takeout. Right. And so in that one symbol, you get a lot of, sort of, secondary meaning. And with fortune cookies, like, it’s technically a cookie, but it also means, like, mysterious, and the future, and the unknown, and like…

Sonal: So, like, sort of primary, secondary meaning. One of the criteria for an emoji to get passed is that it has to have a certain element of ambiguity to it.

Jenny: Well, I think, yeah…

Fred: I love this. I’ve been thinking about this so much. When I did “Emoji Dick,” it was more of an experiment around crowdsourcing an emoji itself. Like, I wasn’t, like, so much interested in making a formal case that emoji could be a language because it was still so early.

Jenny: Yeah, it was very early.

Fred: Could it get there maybe one day? Yeah. But Unicode makes a really good point. They’re like, “Emoji is not a language. It shouldn’t be a language. The value is that it’s ambiguous.” And I’ve really come around to that thinking, and this idea that the charm of sending an emoji is that it can be interpreted in a couple of different ways. And that’s actually why we value it. And I’ll go further and say that — a lot of people ask me why emoji have become so popular. And I think it’s tied to the fact that we now are just inundated with text. We live in a text culture, right? We communicate via text. Our careers are run over email. We read constantly. Everything we do is mediated through almost literal words. And so, emoji represents this kind of reaction to that. And the popularity of emoji, I think, is largely due to the fact that we need some other way of expressing ourselves over text.

Sonal: If the pipes are so mechanical, like, phones and machine, you no longer have the non-verbal aspects.

Fred: Absolutely.

Sonal: So, this is actually replacing sort of this human element of the glimmer in your eye or, like, the blush on your cheek.

Fred: Or even just…

Sonal: There’s an emoji that does that.

Fred: …you think about the amount of signal you get from somebody’s voice on an analog telephone. And when you strip that out and all you’re communicating is, like, LOL, you don’t actually know how sincere that laugh is, or that chuckle, or whatever that person’s trying to convey. And so emoji gives us a much bigger palette to convey this kind of, like, extra, like, limbic meaning that we wanna have in our communications, but we can’t because we’re texting all the time.

Sonal: So, to break down the taxonomy of figural representation not using literal text. Let’s talk about where emoji fits. We have emoticons, which are, like, a colon and a parenthesis, and that gives you a smiley face. Or, like, a semicolon and a parenthesis and that gives you a wink.

Jenny: Right. Using punctuation for existing…

Sonal: Using punctuation is an emoticon.

Jenny: Is often ASCII-ish.

Sonal: Right, because it’s got ASCII art as well.

Fred: And it goes way back. Some of the earliest references to emoticons go back to the 19th century as well, where people…

Jenny: Oh my God.

Fred: Yeah, yeah. People were using colons, and dashes, and parentheses to express, like, a wink. It goes way back. It’s important to add in hieroglyphs and iconography. Other humans have had this idea before. Like, the medium and the technology is kind of, like, incidental.

Sonal: I’m so glad you brought that up, because it’s so important to not get caught up in technology time. Well, technically, technology includes, like, sticks and stones, so that does go back in time. But in the context of this machine web that we live in, then we have emoticons as part of the taxonomy, and then we have emoji. But how would you guys define emoji?

Jenny: It’s Japanese. Drawing language.

Sonal: Emoji.

Jenny: I don’t know how to pronounce [it] in Japanese, but the Chinese — the “emo” is not for emoticon, or emotion or anything. It’s just totally a coincidence.

Sonal: Wow.

Fred: It’s hard not to just hue to the Unicode Standard and say it’s the set of icons defined in Unicode that represent objects, and nouns, and actions and…

Jenny: The way that I explain it to people is, an emoji is a character — an emoji is something you can put in the subject line of an email because it literally is text. So, in the same way that Unicode has, kind of, defined the standard to unify all the graphical representation of different languages throughout the world — and even non-languages, or like, you know, the Wingdings and all of that kind of stuff. Emoji actually slipped into that entire system. So, there is literally what they would call a codepoint assigned to each emoji — or, sorry, not every single one, because now they’re, like, compound emoji. But there are codepoints assigned to emoji, which basically says, “You know, when a computer sees this codepoint, they render it in a certain way.”

Fred: But it’s important to, kind of, wrap your head around what’s actually happening inside the computer, because the emoji is being sent as text. If your computer supports UTF-8, UTF-16 — that’s just like a standard way for your computer to handle text, whether it’s your phone or your laptop — then it’s being told, “Render this emoji.” But it’s actually up to your computer’s operating system, whether it’s OSX or iOS or Android or whatever, to go fish out a little image and put it on your screen. And so that image is actually controlled by the hardware manufacturer or the software manufacturer. You know, when it’s actually rendered on your screen, the operating system is choosing which image to show you. And those images are actually stored, you know, in the same way that other images are stored on your computer as little PNG files. And so, Apple, you know, puts those on your computer, and your computer chooses to render those, which is why you may get slightly different, you know…

Sonal: Different interpretation. Right. I’m glad you walked us a little bit — yeah.

Jenny: And it’s actually really interesting, because recently Facebook just introduced their own emoji and that, like, basically hijack Apple emoji. So, you can turn that on or off, but essentially, they’ll swap out all the ones on the Apple.

Fred: And Twitter has had their own set for a while and so…

Sonal: Why is that? Why do these manufacturers care? Yeah.

Fred: So, there are interesting copyright considerations here. My guess is a lot of those companies are doing it because A, they can afford to make their own set. B, they wanna avoid the legal liability of using Apple’s set.

Sonal: Apple, right.

Fred: And C, like, they think they might kind of have some, like, moment of, like, “Hey, did you see Twitter’s new emoji?” Right? And so, you know, these large companies are kind of…

Sonal: Innovating on emoji.

Fred: Yeah, yeah. Like, re-innovating and re-illustrating their emoji. And I think, you know — I think Microsoft actually just evolved to a new set, or was it Android? I think it might’ve been Google or Android. They just upgraded to make it seem a little bit more normal. Like, they had gone from, like…

Jenny: <crosstalk> the terrible blue and white…

Fred: Yeah, yeah. So…

Jenny: Or, there was, like, the blobby ones that were terrible.

Fred: Yeah, I think Google had blobby ones for a while. And now they’re doing somewhat normal ones.

Jenny: Scariest emoji ever — the Microsoft emoji are, like, blue and gray, and they look like monsters that hide underneath your bed.

Sonal: Why are they blue and gray? Why do they look like that?

Fred: I think it’s just an attempt to be, like, different from, like, the yellow skin tone.

Jenny: Well, also, you have to — part of the original emoji is, you wanted things that were skin tone neutral. So Apple and Google chose yellow, but Microsoft for some reason chose gray.

Sonal: Oh, gray because I was gonna say, for Hindu, like, blue is — actually not a bad thing to have your skin blue. It’s, like, a God.

Jenny: The other thing is, if you have your own set of emoji, you can actually start adding to that set without going through Unicode.

Sonal: Through the Unicode Consortium, right.

Social and political sensitivities

Jenny: So, like, a very good example is the gay family emoji, originally, where it’s not actually one emoji. Another one is, like, man, man, kid, kid. That is actually a compound emoji of four characters glued together using something called a “zero-width joiner,” which is basically like an invisible glue. So, if you are sending that emoji to someone else who doesn’t have the ability to render it out, it actually unravels itself into, like, a multiple character. Now, what you’re seeing is a lot of vendors making compound emojis. And, like, actually one of the places where this is being debated for use is the need for a professional female emoji, right? Because one of the big problems right now, on the existing set of women as represented by emoji is, like, there are only, like, really four roles for women to play, compared to men. You know, men, you can be a sleuth or you can be, you know, a policeman. You can be, sort of, a medical worker…

Fred: Construction worker.

Jenny: There are all kinds of things. You can even be Santa Claus. But as a woman, the four things you can be as a role are, basically, bride, princess, dancer, Playboy bunny.

Sonal: Oh my God.

Jenny: That’s it.

Sonal: It just goes to show you how the — I mean, of course, this is the politics of human life [playing] out in these systems. I mean, the perfect example I was thinking of is the rifle emoji, and the case of, I believe, Apple, Google, and Facebook. Charlie Warzel at Buzzfeed wrote a really detailed article investigating this, and about how they, sort of, helped suppress — as part of the Unicode Consortium — the rifle emoji.

Fred: Right. Emoji already has a gun in it, right? And it’s like, okay, so how many more versions of that do we need? And you’re right, it’s absolutely a political topic. I mean, that issue manifests itself in so many other places than emoji. The country flag stuff is super interesting, because that uses kind of what Jenny’s talking about with these compound emojis. Unicode didn’t actually wanna decide which flags were and weren’t an emoji. So what they did…

Sonal: Right. You’re legitimizing, then, political issues.

Fred: What they did was they built this kind of, like, meta country system, so that you would actually be pairing these country letter emojis together. So CNN would go together, and then it would be up to your phone to decide if you showed the Chinese flag. They pushed that decision-making — that, like, political decision-making of which flags to support — off to the handset manufacturer so…

Jenny: Microsoft actually does something weird there.

Sonal: What do they do?

Jenny: They don’t show a flag. They show a flag plus the two letters.

Fred: The two letters.

Jenny: Microsoft doesn’t render it, like, normally.

Fred: To the point about politics being kind of embedded in emoji, it’s not just because these are icons that represent the parts of our lives that we feel passionate about. It’s because there’s a finite palette. It’s not like language, where you can only — you know, you can kind of combine, say, whatever you want.

Sonal: It’s combinatorial. You can take multiple combinations and turn it into whatever you want.

Fred: Yeah. Language is, like, you get way more degrees of freedom to kind of express yourself. There’s a finite number of food items that are available to go in there. And when you think about the vast, like, multitudes of humanity, whether it’s, you know, people’s relationship status, their sexual orientation, or skin color, it’s like — emoji is never gonna be able to express that. And so, like, how do you contain this thing that’s, like, growing and kind of has to grow as more and more people use it, but also, by definition, has to be a finite list of icons?

Sonal: Well, how do they handle the skin tone issue? Because one of the things that I noticed is that on Apple — because I use an Android so I didn’t notice this — you can press down on a thumbs-up, for example, and then you can pick among 15 different shades to, like, pick a skin code shade that’s closest to you.

Jenny: Five and yellow.

Sonal: Oh, five.

Jenny: Yeah, it’s based on the…

Fred: Do you remember the name Fitzpatrick skin tone scale?

Jenny: Yeah, it’s actually used — it’s the same skin tone system that dermatologists use to categorize.

Sonal: This reminds a little bit of being a kid, when you had [a] Crayola box. I remember that the only shade you had — there was, like, a nude shade, or, like, a skin tone.

Fred: Yeah, and nude was always Caucasian.

Sonal: And I’d use sepia. I remember using sepia to represent my skin color.

Fred: I mean, there’s a great history about this in — this is gonna sound weird for me to say. But, like, women’s pantyhose, like, had this issue where nude was always considered Caucasian, and people were, like, “This is ridiculous.” It was one of the earliest blind spots of emoji I remember. It was like…

Sonal: Right. Well, I mean, if you have, like, only white men designing them. Do you remember when Slack — there was this guy who wrote a post about just the brown hand?

Jenny: Yeah, yeah.

Fred: Yeah.

Sonal: And I remember it was so meaningful, because it’s such a minor seemingly arbitrary thing but then it is true. Like, the first time I saw that I could find my skin color in a system, and to be able to use it, was kind of amazing and empowering. And I think there’s something significant about that.

Fred: I would totally agree. I don’t share your experience as the person on the other side. And so, it’s funny for me because I don’t…

Jenny: He’s a white male, for those of you who cannot see Fred.

Fred: Yeah, so for those of you listening, I’m a white guy. I don’t share that, like, sense of identification with the bright, white, like, <crosstalk> index.

Sonal: Right. You’re like, “That’s not necessarily me.” It’s just, like, a thing.

Fred: Yeah. And I’m like, it feels odd to opt into that, which speaks to my privilege as a white male where I just like…

Sonal: No, it’s not just that. If you’re not exposed to it, you’re not exposed to it. The bottom line is if you’re any person of color, you’re always aware of your color, especially if you’re in a context where everyone else is not the same color as you.

Fred: And so, when I texted my friends who are not white, and I’m like, should I be choosing that one? And I just choose the yellow skin tone.

Jenny: Yellow.

Fred: And that’s just like the — I feel way more comfortable with that.

Sonal: Yeah, yeah.

Jenny: So, my solution is, I often send four. Like, it’ll be, like, yellow, light, dark, and then, like, the beige one.

Sonal: Oh, that’s great.

Jenny: So it’s like — it’s like a Benetton ad in emoji world.

Sonal: Benetton emoji, that’s fabulous.

Fred: So now the, kind of, evolution is that we have yellow for, like, all the human face characters, and then you can choose skin tones for some of them. But it doesn’t get at, like, more nuanced issues about, like, cultural and racial identity having to do with facial structure or hairstyle.

Sonal: Oh, right, the features.

Fred: And these are…

Sonal: That’s a great point, actually, because one of the pet peeves I have is when I used to go to foreign countries and look at billboards, it always glorified that aquiline nose, the face structure — whereas there’s a totally different type of face structure in different areas.

Fred: Emoji probably won’t ever have that amount of, like, customization, and Unicode gets this. And they actually say, like, “We’re adding, like, 60 emoji a year. This is unsustainable. We feel like the future is inline images.” And that, kind of, breaks my heart as, like, kind of a, you know, nerd standardization guy, like, who really appreciates all the hard work that went into Unicode, and the idea that it is a standard. Because if you’re just sending inline images forever, then, like, you know, you have no idea what’s gonna be on the other side and if they can render the image.

Jenny: So stickers. So Kimoji, for example, Kim Kardashian’s “emoji…”

Sonal: It’s awesome.

Jenny: They’re not actually emoji. These are just stickers. They are images that you can text back and forth. But, you know, again, you know, standards — can you put it in the subject line in the email? And those you can’t.

Sonal: You can’t so therefore, they don’t qualify. So just to go back to the…

Jenny: They’re not technically emoji.

Sonal: Right. So then, going back to our hierarchy, we went from emoticon to emoji and now stickers you would define as a…

Jenny: Stickers. Stickers are basically inline images. I mean, stickers are just images that you can pick from a palette.

Fred: And I think you can — you know, in certain apps, you can, like, apply a sticker to an image that it, like, sits on top of it. But you’re then in this, kind of, like, proprietary ecosystem of — that’s okay. But, like, you think about the stuff that really works, and the stuff that really changes the future of the web and communication, and it’s all standardized.

Sonal: It’s all standard — and you’re saying this as a standardization person. Because my friend, Connie, who wrote a wonderful post on the topic of stickers, argues that emoji are very limited for what you need to do, because she feels that you have so much more expression and the ability to convey so much more with stickers than you do with emoji.

Jenny: Emoji doesn’t preclude the use of stickers. There are some sets of images that are universal enough that should be hard-wired into the operating systems, and basically can be cross-platformed that an iOS device can talk to — you know, Microsoft Windows can talk to, like, an Android device, can talk to your Mac laptop. Like, the fact that — at least you’re not gonna get little square boxes as long as your operating systems are fairly up-to-date.

Sonal: Well, that goes, then, to your point about why standardization is important, because you’re now giving up that you’re in this proprietary ecosystem like WeChat or Line, and you only have their sticker set. And you can’t always transfer all these stickers across…

Fred: And also, if you think about the accessibility issues around stickers, right? Like, people using screen readers— they’re not gonna be able to interpret an image. And, like, emoji actually have names. And so, in theory, there’s much better accessibility for emoji for somebody who’s visually impaired, so.

Jenny: Yeah. Like, for example, last year, Oxford English Dictionary chose “face with tears of joy.”

Fred: “Face with tears of joy,” yeah.

Jenny: Which I always thought looked very sad.

Fred: Yeah, it’s…

Jenny: You know, the thing with the eyes and it’s, like, bawling. But that’s actually “face [with] tears of joy.” And you know that because, you know, all these emoji have…

Sonal: They say the label — Oxford put that in there.

Fred: That was the word of the year.

Jenny: So, the word of the year was an emoji.

Fred: Part of the reason they chose that was that it ended up as number one on my friend’s site, called emojitracker.com.

Sonal: Oh, right. That’s right. The emoji tracker, which tracks all the use of emoji on Twitter.

Fred: And for a while, it was just, like — it was, like, the heart emoji or something, or just the smiling face emoji. So, I think it’s really interesting when the top emoji shuffle, because, you know, whenever you start texting with somebody who hasn’t used emoji before, they’re, like, choosing, like, the safest ones.

Bitmoji and expressing emotion in text

Sonal: Going back to this idea of some of the companies owning their own emoji, and some of the proprietary open tension between standardization, freedom of expression — what do you make of this notion that part of what we’re doing here is essentially also creating a more machine-readable web, in terms of emotional reading? Because, essentially, you’re now adding a whole new layer where you can codify people’s emotion, sentiment — in ways beyond just a black and white, like, don’t like.

Fred: I’ve been thinking about this so much, actually, and not in the context of emoji, but actually Facebook reactions.

Sonal: Yeah, me too. I used to assign and edit op-eds on this topic because I was very obsessed with it.

Fred: I think it’s a really interesting topic because if you look at traditional sentiment analysis in the data world, it’s kind of a joke. You have to have training data, you have to know good cases. And the…

Sonal: And just to interject for a moment, as someone who’s tested a million of those systems and can never find one that actually works for my needs, they are so binary. You don’t get anything useful, and you’re not getting insight.

Fred: One of the reasons there is that words have these degrees of freedom. They can be used sarcastically, and you would never know it based on the semantics. And so, traditional sentiment analysis is really broken, because you’re using these, kind of, like, stale, rigid semantic definitions. What’s really interesting about Facebook reactions is, you know, you think you’re saying, “I love this thing,” or, “I’m sad about this,” or, “I’m angry about this.” But what you’re actually doing, in conjunction with that, is giving Facebook really great labeled data for sentiment analysis.

Sonal: That’s right. Machine-readable data. That is a holy grail of emotional sentiment understanding. When I was at WIRED, I assigned a piece to a sociologist, Evan Selinger, because I wanted to coin this phrase — the mood graph — because we have an interest graph, social graph, you know, all kinds of other graphs that link all these nodes and ideas. And now, to have, like, a mood graph, to essentially be able to put your pulse on someone’s mood — something very finite, yet constantly changing. It’s just a fascinating thing to be able to codify this.

Jenny: The sentiment stuff generally correlates very strongly with [the] human face and body. So I think this is also why people agitate so much for emoji that look like themselves. Like the redheads, and people with beards, and people, you know, who are bald.

Sonal: Or anyone who has curly hair. People with curly hair relate to other people with curly hair.

Jenny: And so, I think people really love seeing themselves represented in emoji, which is why Bitmoji, which is highly, highly, highly customized stickers in sort of emoji spirit.,,

Sonal: Oh, my cousins and I Bitmoji on WhatsApp all the time. I think there’s something really symbolically important about Bitmoji, because you are putting yourself in it and conveying in this sticker form. The fact that Snapchat bought it I think is really telling.

Jenny: Oh, yeah, for $100 million. Is that right?

Sonal: Right. Especially given that they are changing this culture of how you express yourself through your facial expressions, with face swapping and filters. Connie and I made the argument that it’s sort of like a new — like selfies. It’s selfies as a form of stickers. So what we’re talking about, with the machine-readable, is a little [more] distinct than this, but it’s sort of an interesting idea all the same.

Fred: I also think it ties into this slightly dubious notion of the uncanny valley where if you wanna try to represent yourself and you wanna have, like, configurability around that, it needs to be, kind of, cartoonish for it to be believable. I think what we’re seeing with Snapchat filters — and I don’t know if you guys have played with SNOW yet. That’s like a…

Sonal: No, I haven’t.

Fred: It’s, like, take Snapchat filters and just multiply them by a thousand. It’s just, like, amazing amounts of diversity around the amount of stuff you can put on your face. It is this weird convergence on identity and emoji that’s kind of happening.

Sonal: I agree and, in fact — this is gonna sound, like, a little out of left field for a moment — but the whole notion around the Chewbacca mask lady, when — you know, that was the most popular Facebook live video ever. It got, like, unprecedented views, and it was simply a woman who was trying on her Chewbacca mask in the car. And she’s laughing and giggling about it. And then she puts her mask on and then she takes it off, and she laughs so uninhibitedly, it’s insane. And I make the argument that what was so empowering — because it took off for obvious reasons — is not the fact that she was laughing so uninhibitedly. It’s the fact that it took putting on and then taking off the mask for her to do that, which is not unlike what happens with communication through these filters, and being able to now express yourself through these cartoon-like ways in a real way.

Fred: I mean, honestly, it takes me back to, like, theater, and, like, Shakespeare in, like, seventh and eighth grade. I remember having these, like, really intense discussions about, like, what it is to put on a mask and what a mask represents about yourself.

Sonal: It’s a very Campbellian idea, right? The Joseph Campbell, like, mask, and the myth, and the man. You’re right. There’s a theater — I mean, that’s why people say improv is so interesting for any career field, but I think that there is an interesting moment now coming together with selfie stickers, emoji, Bitmojis altogether, where we do have this new emotional web coming together.

Fred: Right. And using emoji — the first time I thought about this — could be kind of, like, putting on a mask over your, you know, self to…

Sonal: Words.

Fred: Yeah, over your words to convey to yourself this, like, this extra this kind of additional layer, this emphasis of your emotion that you otherwise might not get.

Creating “Emoji Dick”

Sonal: Okay. So, going back to you writing an entire book in emoji, and yet you were saying that you’ve kind of evolved in your thinking, you know, that emoji is not necessarily a language but clearly, it is a visual language. And it is a tool for communication. It’s not complete. So, how did you translate that? I mean, what were some of the trade-offs and decisions you made? And, by the way, for the audience — that book was, like, 2009 or that was, like, many years ago.

Fred: Okay. Okay. I’ll…

Sonal: So what emojis base were you working off? Did you make them up? Like, what’d you do?

Fred: So, I’d gotten a text from my college roommate whose wife is Japanese. He sent me an emoji, and I was like, “What is that?” They told me you could download, like, basically a Japanese app and it would, like, awaken your iPhone to the emoji keyboard, like…

Jenny: Come alive, emoji.

Fred: It just spoke to me in the, like, you have to hack the iPhone to get the special keyboard of, like, Japanese icons. And I was like, “Oh my God, I want this so bad.” I was like, “This is amazing. I should write a book in emoji.” And I was like, “Oh, that’s a lot of work. I don’t know if I can write a whole book in emoji.” And then I was like, “Or maybe I can translate a book in emoji.” I was like, “Okay, what books would work?” And I was like, “Well, it has to be in the public domain,” because I worked a lot in, like, the copyright reform space. Nobody is gonna just, like, let me translate their book into emoji without a lot of effort. For a moment I thought about the Bible. And I was, like, that’s too obvious. What’s, like, totally, even more inappropriate?

Sonal: So “Moby Dick” came to mind.

Fred: Yeah, it came to mind as, like, this, like, impossible book to put into these symbolic characters. As soon as I thought I was like, “No, I can’t do that. That’s crazy.” And I was like, “That’s, like, too hard.”

Sonal: Honestly, it’s a little bit like — I just came back from seeing “Hamilton.” And so, it’s a little bit like the idea of putting a rap to, like, the founding fathers. That’s what I find so fascinating.

Fred: Yeah, I would say “Hamilton” was probably…

Sonal: It’s like a mashup of mediums, and time, and culture.

Fred: And it’s like one of those things where you tell it to somebody and they’re like, “You can’t do that. That’s crazy.” And then you’re like, “Well, the fact that you just said that made me wanna do it.” And so…

Jenny: And not only that. There are not one but two whale emoji. Were there at that time?

Fred: No, there was only the original…

Jenny: The cute one?

Fred: The cute one. The, kind of, <inaudible> style one.

Jenny: Cute one, aww. So, there was a whale emoji.

Sonal: What’s his name? Ahab is battling the cute whale. Aww.

Fred: Yeah.

Jenny: That’s the second one.

Fred: Yeah, I think it’s called sperm whale — didn’t come up until later. So, I was like, okay, wow. That would be really interesting to do all of “Moby Dick,” because it’s also, like, really long. I mean, it’s 10,000 sentences. And okay, well, if I don’t wanna do this, maybe I can hire somebody to do this. And I was, like, experimenting with Mechanical Turk at the same time. I think it was, like, one of the original Amazon Web Services. It was, like, it would later become, you know, part of that AWS umbrella.

Sonal: Yeah, I remember people using it for research and stuff.

Fred: Right. It’s still used for research. It’s still invaluable for that. But, you know, a couple of other people had done, like, an experiment here or there, like, using it, like, off-label. I had made a task at Mechanical Turk — just to ask Turk workers, “If you could ask anyone, like, to do anything on Mechanical Turk, what would you have them do?” And they came up with this long list of stuff. And I don’t think “translate a book into emoji” was one of them. But there was some creativity out there. I was, like, okay, I’m gonna try this thing where I’m gonna hire people to translate “Moby Dick” into emoji, some portion of it, and see if this works. So, I did the first chapter and the results came back, and they were hilarious. They were so good.

Sonal: They were good.

Fred: Yeah, they were great.

Sonal: How did you assess that? First of all, what do you mean you did the first chapter? Like, did they break it down word by word? How do you capture that in emoji? Is it like a sentence?

Fred: So, I decided I was gonna do it on a per sentence basis. And that actually turned out to be one of the challenging parts of the project was, like — splicing sentences is actually like, kind of, like, a classically hard and a natural language processing problem. And so I kinda, like, figured out a hack to, like, chop it up. And I wrote a lot of regular expressions to basically get the whole book into sentences.

Sonal: Wow. But you decided basically the sentence was the unit of analysis, not a phrase, not a word, a sentence.

Fred: You would have a sentence in the task and you’d say, “Pick any of these emoji.” And then I actually wrote my own little emoji picker, because these things didn’t exist at the time. I had gotten the emoji from a friend. He had reverse engineered the iPhone SDK and basically hacked out the PNG files from the software kit to basically have the raw emoji in image form. And so, I took that and just made, like, a little JavaScript, like, HTML thing and, you know, dumped that into Mechanical Turk. And it came back and I was like, hey, this works. And so, I think the sentence that’s kind of, like, on the cover of the book if you go to the website, it’s, like…

Sonal: The website being Emoji Dick.

Fred: emojidick.com. “Call me Ishmael” is the first sentence of “Moby Dick.” And the emoji that the Turk worker chose was, like, telephone, man with face, sailboat, whale emoji. It’s perfect.

Sonal: That’s amazing.

Fred: That was just like — but the rest of it was just, like, indecipherable emoji nonsense. And some of the people were just, like, all right, “Give me my five cents. I’m gonna click some random emoji.” And other people just, like, clicked every single emoji. So, the plan became — have people translate the same sentence multiple times. So, you get three different emoji translations for one sentence. And then have another set of tasks where people vote on the best, most appropriate translation. So, like, of the three, which one got the meaning across the best? And I was, like, oh, I was just, like, getting really excited about this. And I started doing the math on how much it was gonna cost. And it was, like, oh, it’s gonna be thousands and thousands of dollars. That summer, I met the Kickstarter guys. I started talking with Andy Baio. He was like, “You should put it on Kickstarter.” So that night I went home and put it on Kickstarter, launched it the next day, and ended up working for them and…

Sonal: And, by the way, how much money did the campaign make?

Fred: My goal was, like, $3,500. I ended up raising $3,700. So I worked on it for, you know, nights and weekends for another, like, eight or nine months. And then, you know, self-published it on lulu.com. You can still buy it. It gets printed on demand. And, you know…

Jenny: Do people still buy it?

Fred: I’ve sold, like, thousands of dollars of “Emoji Dick.” And I’d say hundreds of copies. And probably, like, 500 or 600 copies of it have sold since then, which is not a lot.

Jenny: I bet this podcast is gonna sell a bunch.

Fred: Yeah, well…

Jenny: You better share some of the proceeds with me.

Fred: Okay, so there are two copies. There’s the black and white copy, which is, like, the easy to print one, and that’s, like, $20 or $30. And then there’s the full color one, which, like, is obviously preferable, because emoji are so colorful. But when you’re printing on demand, 800 pages of color laser hardbound copy — it’s actually really expensive. So, that thing costs, like, $180.

Sonal: Right, because you’re not printing in bulk.

Fred: Exactly.

Sonal: Because you actually save money when you print in bulk, right.

Fred: So I have to sell that one for that much.

Sonal: Damn.

Fred: And, like, people still buy it. In 2013, The Library of Congress contacted me and, you know, they said, “We would like to acquire ‘Emoji Dick’ as our first emoji book.” I was like, “Are you sure?” They’re like, “Yeah, yeah, we’re sure.” And I was telling a friend — and David Gallagher, I think you must know from the Times. And he’s like, you know, everyone submits their stuff to the Library of Congress. It’s not that big of a deal. And I was like, “No, man, they asked for it, like, they’re acquiring it.”

Sonal: I think it’s a big deal because it was a curatorial point of view.

Fred: Totally.

Sonal: They’re saying, “This is a cultural moment. It’s not just a book that was published, and we need to figure out how to acquire it.”

Fred: I was like, “All right. I’ll spare a copy.” I signed it. I sent it to them. And then they sent me this little, like, you know, certificate in digital form. It was hilarious — and this is my favorite part — is that it’s somehow listed as a translation of “Moby Dick.” So when you look up “Emoji Dick,” it says all these libraries have it, because it’s really just saying that, like, they have a translation. They have the original “Moby Dick.” Now it’s got a life of its own, and people still discover it, and yeah.

Sonal: That’s amazing. I mean, you actually even curated an art show, didn’t you, based on this?

Fred: Yeah. Friends of mine put together a kind of emoji survey art show, and there [was] some really great stuff in there. Emojitracker was there. There was a programming language built out of emoji. There was a lot of other good stuff.

Jenny: I mean, emojis can have their URL. I mean, that’s another thing. They’re literally text, so you can have, like, emoji@…well, I don’t know, @gmail. But you can have emoji in your email address.

Sonal: Oh, you can?

Jenny: Totally.

Fred: You can also buy emoji domains.

Sonal: So you have an emoji book. You have emoji art shows…

Jenny: Emoji hackathons.

Sonal: Emoji hackathons.

Jenny: So, our big news this week is that in November, in San Francisco, we are going to throw the first-ever Emojicon, which is basically…

Sonal: What? Is that like Comic-Con?

Jenny: It’s like Comi-Con, but emoji, of emoji.

Fred: I really hope people show up dressed in emoji costumes.

Sonal: I was about to say, I’m gonna show up as — you guys are gonna — Yiying is gonna show up as dumpling emoji for sure.

Jenny: Or, like, poop emoji or, like, the ghost emoji. So it has many different elements to it. So one is definitely, sort of, this whole “emoji learn” aspect, where it’s, like, panels and talks. And there’s, sort of, emoji film festival, and there’s an emoji hackathon, and then there’s an emoji art show. And then, of course, the opening party emoji where, you know, our goal is to only have food that is also emoji.

Sonal: So, why a conference? I mean, of course, I see the cultural significance, but to bring people together around this idea of a first-ever Emojicon, like, what’s the significance of that?

Jenny: Part of it was, I thought it already existed. And to me, the fact it didn’t…

Sonal: I kind of did too, to be honest. When you just said that, I was like, what?

Jenny: Yeah. And then I was like, the fact it didn’t exist — and I kind of have this issue where of, like, if I think something needs to be — I will try to make it exist.

Sonal: You will make it exist, God damn it.

Jenny: Right. So, we did it with dumpling emoji. We did it with Emojicon. And so we actually have some really cool sponsors. We’re gonna have a lot of, kind of, emoji activists, kind of, out there.

Sonal: Emoji activists, I love that.

Jenny: And also, from our perspective — you know, there are a lot of policy decisions around emoji and, obviously, the world really cares about emoji. Whether or not it’s the rifle emoji, or the condom emoji, or, like, professional women emoji. Part of the goal of Emojicon is to open up that discussion, so it does not just held at the Unicode level so to…

Sonal: So, are Unicode members gonna be attending this conference?

Jenny: Oh, members of Unicode Emoji Subcommittee including, like, you know, the co-chairs. And we’ve timed it in November between the Unicode Conference itself and the Unicode Technical Committee meeting. And also, like, it’s right around election day.

Sonal: Well, you guys, thank you for joining the “CFI Podcast.”

Fred: Thanks for having us. This was so much fun.

Sonal: This was so much fun. We could keep going…

Jenny: So much fun, hours and hours on emoji non-stop.

Sonal: Yeah, I wish we could.