## Instagram Demo: Your friends are more popular than you

I’m teaching a class that uses code to discover unintuitive things about social systems (UC Davis’ CMN 151). One great one shows how hard it is to think about social networks, and it’s easy to state: “On average, your friends are more popular than you” (Feld 1991).

It’s one thing to explain, but something more to show it. I had a demo coded up on Facebook, but it was super fragile, and more of my students use Instagram anyway, so I coded it up again.

To run the demo you

1. Consider not to participating (because, for a student, the demo involves logging into your Instagram account on a public computer and running code written by someone with power over you).
3. Click to show your Followers, and scroll down that list all the way until they are all showing. This could take a while for people with many followers.
4. Open up View -> Developer -> JavaScript Console (in Chrome. “Web Console” in Firefox. Slightly different for other browsers. In Safari you need to find developer mode first and turn it on)
5. Ask them to paste the code below, which will be accessible to them via Canvas, into their browser’s JavaScript Console. If Followers aren’t showing, it won’t work. This could also take a while if you have many followers. Keep pasting the last part until the numbers are stable. You computer is working in the background growing the list of your followers’ numbers of followers.
7. Paste your values into the sheet.
8. Calculate the average number of followers, and the average number of followers of followers. Compare them. With enough participants, the second will be bigger, even if you exclude giant robot accounts.

This post isn’t an explainer, so I won’t get into how and why it’s true. But the way you set it up beforehand in class is by reasoning that there shouldn’t be a systematic difference between your and your friends’ popularities. The numbers should be the same. You wrap the lesson up after the data is in by hopping onto the spreadsheet live and coding up the averages of their followers, and of their friends followers, to show that their friends’ average is higher on averages. After explaining about fat tails, you drive it home on the board by drawing a star-shaped network and showing that the central node is the only one that is more popular than her friends, and all others are less popular.

## The code

Open your Instagram Followers (so that the URL in the location bar reads https://www.instagram.com/yourusername/followers/) and paste this into your JavaScript console.

 // from https://stackoverflow.com/questions/951021/what-is-the-javascript-version-of-sleep function sleep(ms) { return new Promise(resolve => setTimeout(resolve, ms)); } function instaFollowerCount(page) { return parseInt(page.querySelector("a[href$='/followers/']").firstElementChild.textContent.replace(/,/g, "")) } function instaFollowerCount2(page) { return parseInt(page.querySelector("head meta[name='description']").attributes['content'].value.match(/([\d,]+)\sFollowers/)[1].replace(/,/g, "") ) } function instaFollowerList(page) { return Array.prototype.slice.call(page.querySelector("div[role='presentation'] div[role='dialog']").querySelector("ul").querySelectorAll("a[title]")).map(x => x.href) } // https://stackoverflow.com/questions/247483/http-get-request-in-javascript#4033310 function httpGet(theUrl) { var xmlHttp = new XMLHttpRequest(); xmlHttp.responseType = 'document'; xmlHttp.open( "GET", theUrl, false ); // false for synchronous request xmlHttp.send( null ); return xmlHttp.response; } function httpGetAsync(theUrl, callback) { var xmlHttp = new XMLHttpRequest(); xmlHttp.responseType = 'document'; xmlHttp.onreadystatechange = function() { if (xmlHttp.readyState == 4 && xmlHttp.status == 200) callback(xmlHttp.response); } xmlHttp.open("GET", theUrl, true); // true for asynchronous xmlHttp.send(null); } var iFollowers = instaFollowerCount(document); var aFollowers = instaFollowerList(document); var docs = []; for (f in aFollowers) { httpGetAsync(aFollowers[f] + "followers/", function(response) { docs.push(instaFollowerCount2(response)); }); if(f % 100 == 0 & f > 0) { await sleep( 1000 * 60 * 30 + 10000); // in ms, so 1000 = 1 second. // instagram limits you to 200 queries per hour, so this institutes a 30 minute (plus wiggle) wait every 100 queries // If you're fine running the demo with just a sample of 200 of your followers, that should be fine, and it's also way faster: this demo can run in seconds instead of taking all night. To have it that way, delete the above 'await sleep' line.   } }  And then, after waiting until docs.length is close enough to iFollowers, run  console.log(You have${iFollowers} followers); console.log((You've heard from ${docs.length} of them)); console.log(""); console.log(On average, they have${docs.reduce((total, val, i, arr) => total + val) / docs.length} followers); console.log(Your most popular follower has ${docs.reduce((incumbent, challenger, i, arr) => incumbent > challenger ? incumbent : challenger)} followers); console.log(Your least popular follower has${docs.reduce((incumbent, challenger, i, arr) => incumbent < challenger ? incumbent : challenger)} followers); 

The result isn't meaningful for just one person, but with enough people, it's a strong lively demo. See how things are coming along for others on this Sheet.

### Technical details

Instagram crippled their API, so it isn't possible to run this demo above board, not even with the /self functionality, which should be enough since all participants are logged in to their own accounts. This code works by getting the list of usernames of all followers and posting a GET request for that users page. But Instagram can tell you are scraping so it cripples the response. That's why instaFollowerCount differs from instaFollowerCount2. In the main user's page, the followers are prominent and relatively easy to scrape, but the requested page of the friend can't be reached through a console request. Fortunately, Instagram's "meta" summary description of a user's page in the lists their number of followers, so a simple regex yields it. Of course, even scraping the follower count and IDs from the main page is tricky because Instagram has some scheme to scramble all class names for every page load or account or something. Fortunately it's still a semantic layout, so selector queries for semantic attributes like "content", "description", and "presentation" work just fine to dig up the right elements. Of course, this could all change tomorrow: I have no idea how robust this code is, but it works on Oct 24, 2018. Let me know if your mileage varies.

## Change your baby’s astrological sign with physics!

My summer project this year was a little non-academic web app project.

http://whatsyoursign.baby/

The premise of the site is that the mechanism of astrology is gravitational influence, and that since small nearby things have influence comparable to large things far away, it should be possible to tune your child’s astrological sign by giving birth around specifically arranged person-made objects. As a pop science site, you’ll see that it is a pretty soft sell: not telling anyone that astrology is wrong, instead trying to channel the interest in astrology into relevant subjects of physics.

I haven’t even released the site yet, but as a summer project it’s already a big success. I developed my frontend skills a bunch, and learned how to use astrological ephemeris databases. I also learned that astrology has a big open source community. I learned that there is a .baby and .amazon top-level domain for web addresses. I also learned a bit more about how to teach web programming students, hopefully showing the bones of the Internet a bit and making code a bit less intimidating.

## Do you lose things? Here’s the magical way to find them.

Let’s say you make a trip to the store, making sure to lock the door behind your on the way out. When you return and try to let yourself in, you discover that you lost your keys somewhere along the way. Round-trip, the whole distance traveled was longish to hunt for a pair of lost keys, like 1km. They could be anywhere!

How should you go about finding your keys? Should you spend the whole cold day and night slowly scouring your path? That sounds awful. But reality isn’t going to do you any favors: there’s no way your keys are more likely to be in one place along the way than another. So, for example, if the space within ten meters of your door accounts for 2% of the whole trip, the probability of finding your keys within that space must be equal to 2%, not greater than or less than 2%. Right?

Nope. It turns out that reality wants to do you a favor. There’s a good place to look for your keys.

Intuition says that they are as likely to be in one place along the way as any other. And intuition is right for the special case that your keys were definitely very secure and very unlikely to have fallen out on that particular trip. But they probably weren’t. After all, if it was so unlikely, they shouldn’t have fallen out. So we can’t just consider the world where the very unlikely happened. We have to consider several possible worlds of two rough types:
* The worlds in which your keys were very secure, but the very unlikely happened and they fell out anyway.
* The worlds in which your keys, on that particular trip, were unusually loose and bound to fall out.
So those are the two types of possible world we’re in, and we don’t have to consider them equally. The mere fact that your keys fell out means it’s more likely that you’re in the second type of world, that they were bound to fall out. And if they were bound to fall out, then they probably fell out right away. Why? We can take those worlds and divide them again, into those where your keys were likely but not too too likely to fall out, and those in which your keys were not just very likely, but especially very likely to fall out. And so on. Of the worlds in which your keys were bound to fall out, the ones that are most likely are the ones in which they fell out right away.

So there it is. If you lost your keys somewhere along a long stretch, you don’t have to search every bit of it equally, because they most likely fell out on your way down the doorstep, or thereabouts. The probability of finding your keys within 10 meters of the door is greater than 2%, possibly much greater.

What is the probability exactly? If you’d had several keys to lose, we might be able to better estimate which specific world we’re in of the millions. But even with just one key lost, the mere fact that it got lost means it was most likely to have gotten lost immediately.

### Why is it magic?

If you know the likelihood of losing your keys, that makes them impossible to find. If you have no idea the chances they fell out, then they’re more than likely near the door. It’s your uncertainty about how you lost them that causes them to be easy to find. It’s as if the Universe is saying “Aww, here you go, you pitiful ignorant thing.”

### Solving the puzzle, with and without data

So you can’t get the actual probability without estimates of how often this trick works.  But even without hard data, we can still describe the general pattern. The math behind this is tractable, in that someone who knows how to prove things can show that the distribution of your key over the length of the route follows an exponential distribution, not a uniform distribution, with most of the probability mass near the starting point, and a smooth falling off as you get further away. The exponential distribution is commonly used for describing waiting times between events that are increasingly likely to have happened at least once as time goes by. Here is my physicist friend, “quantitative epistemologist” Damian Sowinski explaining how it is that your uncertainty about the world causes the world to put your keys close to your door.

If you get in this situation and try this trick, write me whether it worked or not and I’ll keep a record that we can use to solve for lambda in Damian’s notes.

In the meantime, we do have one real-world data point. This all happened to me recently on my way to and from the gym. I was panicking until I realized that if they fell out at all, they probably fell out right away. And like magic, I checked around my starting point And There They Were. It’s an absolutely magical feeling when mere logic helps you solve a real problem in the real world. I’ve never been so happy to have lost my keys.

### UPDATE: How strong is the effect?

All of the above tells us that there’s a better than 2% chance of finding your keys in the first 10 meters. But how much better than 2%?  20% or 2.001%?  If the latter, then we’re really talking intellectual interest more than a pro-tip; even if the universe is doing you a favor, it’s not exactly bending over backwards for you.  To tackle this, we have mathematician Austin Shapiro.  Backing him up I can add that, on the occasion on which this trick worked for me, my keys were super super loose, just like he predicts.  A takeaway is going to be that if this trick works for you, you did a very bad job of securing your keys.

I read your blog post, including Damian’s note. I have some things to add, but to clearly explain where they fit in, let me try to delineate two separate “chapters” in the solution to your key problem.

In chapter 1, we narrow our set of models for the location of the keys to the exponential distributions. Damian gives a good account of how this can be justified from first principles. But after doing this, we still have an infinite set of models, because an exponential distribution depends on a parameter $\dpi{300}\inline \lambda$ (the expected rate of key losses per kilometer walked, which may be high if the keys are loose and hanging out of your pocket, or low if they are well secured).

In chapter 2, we use conditional probability to select among the possible values of $\dpi{300}\inline \lambda$, or, as you put it in your blog post, try to figure out which world we are in. This is the part that interests me, and it’s also the part that still needs mathematical fleshing-out. All Damian says about it is “So what is the value of $\dpi{300}\inline \lambda$? That’s a question for experiment — one must measure it.” But as you say, we’ve already done one experiment: you observed that your keys did fall out during a 1 km walk. This is enough to put a posterior distribution on $\dpi{300}\inline \lambda$if we posit a prior distribution.

However… what does a neutral prior for $\dpi{300}\inline \lambda$ look like? I don’t know any principled way to choose. A uniform distribution between 0 and some finite ceiling is unsuitable, since according to such a model, if you’re ever very likely to lose your keys, you’re usually pretty likely to lose your keys.

Assigning $\dpi{300}\inline \lambda$ itself an exponential prior distribution seems murkily more realistic, so I tried that. If $\dpi{300}\inline \lambda\sim{\rm Exp}(k)$, then, if I did my math right, your probability of having lost your keys in the first $\dpi{300}\inline x$ km of your walk works out to $\dpi{300}\inline k(k+1)\left(\frac 1k-\frac 1{k+x}\right)$, which is $\dpi{300}\inline (1+\frac 1k)x+O(x^2)$ for small $\dpi{300}\inline x$. So in this case, Bayesian reasoning boosts the chances that you lost your keys in the first, say, 10 meters, by a factor of $\dpi{300}\inline 1+\frac 1k$. Observe that for this effect to be large, $\dpi{300}\inline k$ has to be pretty small… and the smaller $\dpi{300}\inline k$ is, the higher your average propensity to lose your keys (the mean of the exponential distribution is $\dpi{300}\inline \frac 1k$). Thus, for example, to achieve the result that the universe is helping you find your keys to the tune of a factor of 5 — i.e., that your chance of having lost your keys in the first 10 meters is 5% instead of the “intuitive” 1% — you need to assume that, a priori, you’re so careless with your keys as to lose them 4 times per kilometer on an average trip. That prior seems just as implausible as the uniform prior.

I can think of one kind of prior that could lead to a strong finding that the universe wants to help you find your keys. That would be a bimodal prior, with a high probability that $\dpi{300}\inline \lambda$ is close to 0 (key chained to both nipple rings) and a small probability that $\dpi{300}\inline \lambda$ is very large (key scotch-taped to beard), with nothing in between. But I can’t think of any reason to posit such a prior that isn’t transparently circular reasoning, motivated by the answer we’re trying to prove.

So… while all the exponential models definitely give you a better chance of finding your keys near the beginning of your route than near the end, I’m not convinced the effect size is all that strong; or, if it is (and you do have one magical experience to suggest it is), I’m not convinced that math is the reason!

Au.

## Burning plane falling out of the sky

Question: What’s wrong with this picture?

Look at the spirals on the engine. Those are only visible when the turbine isn’t spinning. Bad photoshop.

This entry was posted on Thursday, July 16th, 2015 and is filed under tricks.

## Gondry video in reverse

Love Gondry’s music videos but hate riding trains facing backward? This is for you:

Original

This entry was posted on Sunday, May 31st, 2015 and is filed under tricks, videos.

## Design for magical spherical dice (3D printed)

I designed a die. It’s special because it’s a sphere pretending to have six sides: each roll will end with one to six dots facing up. It’s also special because you can print a copy. The trick is a weight that falls into one of six pockets under each of the numbers. “Spherical dice” sounds better than “spherical die,” so print two.

Some assembly is required: You just have to drop in the weight and jam in the plug. According to the colleague that helped me, Nobuyuki Umetani, fishing stores are the best place to get lead. For the plug, you can use your thumb. Most of the plug will still be sticking out post-jam and you’ll have to snap off the rest. The way many printers print makes them snap clean along the path of the printer head. So score the plug by drawing a knife around it’s diameter where it meets the sphere, steady it (with a vice or on the edge of a table), and give it a good whack.

### Notes

• The roll has satisfying action. Video at bottom.
• The plug is tapered so as to jam well. It functions as the number one.
• With this design, the strength of the jam may influence the fairness of the die. Probably not a real concern (since the ball’s mid-air choice of pockets will have a bigger influence on the outcome), but this is an imperfection in the design. Someone will have to do a few thousand or so rolls to make sure.
• The density of the fill and the weight of the missing dots could also influence a die’s fairness, but if you care that much then you know not to bet six with any dice that didn’t come through a casino.
• You can fill the dots in if you want them to stand out. Nail polish will do. Just be careful: the plastic doesn’t forgive mistakes because its layers act like capillaries and suck up liquidy paint (or nail polish removered nail polish)
• You want the diameter of the lead weight to leave some wiggle in the pockets. If your weight is a snug fit into the die, get a smaller weight (or scale up the size of the model).
• I’ve oriented the model at an angle so that it’ll print correctly (without any support material on the inside) if your printer can handle printing a 45° overhang. It probably can? I don’t know how common that is, but the machine I used can.
• The original design subtracted an octahedron from the center of the sphere, but it was a little too sensitive, and also harder to make fair, so I redesigned it to subtract three mutually orthogonal boxes.
• Workflow was 123D (for the orthogonal part) to Meshmixer (to sculpt out the dots) to MakerWare (staging and path planning) to a second generation Replicator.
• I got the idea from someone who did the same thing in wood. I saw it for sale at a store call Aha.

And, this is how I roll:

This entry was posted on Saturday, July 26th, 2014 and is filed under straight-geek, tricks.

I used to find myself on Facebook even when I did not want to be there. Now that doesn’t happen any more. Every time I go I have to click through a bunch of popups. The number of popups increases by one each time I return. I can still check the site, it just takes a little work, and a little more work each time.

With the carefully engineered convenience of these sites, you can reach a point where spasms of muscle memory override your own intentions about where you want your mind. If you think a small simple barrier would help you be a more mindful consumer of social media, you can install an extension I made for Chrome.

Even if you check the “do not show popups” box every time, this plugin will still force at least three clicks before every page access. And it will still make it easier to stop than to continue. And it will still keep count.

### nerds

Here is the early code (you can ask me for more recent code). And these pages are useful for authoring.

This entry was posted on Monday, May 19th, 2014 and is filed under straight-geek, tricks.

## How to learn every spice in the cabinet

So many of my peers are going epicurean. Its beautiful because I think cooking is empowering: it encourages people to try new things and experiencing new ways of thinking. It worrisome because it provides another thoroughly commodified identity, with all kinds of vocabulary for justifying not liking something. I spent a year convinced that I knew how I liked my coffee (and that I didn’t like it any other way). I finally admitted to myself that it was a delusion, and that the variance between cups of coffee was greater than my ability to tell the difference.

So we’ll focus on the first: empowerment. I never knew how to use the spices. I figured that the best way was to just cook lots of recipes by the book until I got the hang of it. But so much of the joy of cooking, for me, is making stuff up. For a while I just cooked without spices at all. I still prefer it that way, but I wanted to learn the spices, so I shifted to throwing in tons of random everything. Occasionally I would make things that worked. While the random approach will eventually start to pay off fine, it requires a certain affection for failure.

Though my stance towards failure is particularly affectionate, I did eventually refine my technique, and now it is fancy enough for anyone to learn to use any spice. Did you know that taste and smell are thoroughly integrated senses? Did you know that the tastiness of coffee and chocolate is entirely illusory? Coffee and chocolate have no taste. They are entirely smell. Try eating chocolate while holding your nose: all you’ll taste is the added sugar. And you can use this confound of the senses to simulate the experience of a new spice without committing to it. Soup is the easiest for this technique, so I’ll focus on it, but it works for everything:

Make your soup without any spices at all, throwing in all kind of stuff and putting off any spicing towards the end. When you are ready, ladle a little thimbleful of soup into a cup and walk over to you spice cabinet. Now open a random spice, take a sip of the soup, and smell the spice, and the next spice, and on down the line. By mixing smell and taste, you can simulate the experience of the soup with the spice. If you like what you are tasting, add the spice to the soup, erring on the side of too little. You can try all kinds of exotic spices and figure out what you like with impunity. Its simple and intuitive, and it will eventually get you a familiarity with the spice cabinet that you didn’t imagine yourself capable of. Feel the power of spice through your main course!