Why Google Stadia is Failing (and will Fail)

Note: I started writing this article about a year or two ago. Today I was browsing through my drafts and I found it. Stadia was recently shut down, so I was right. The article remains unfinished, and begins here.

Latency.

There. that’s the article.

No, but seriously. Latency is a deal-breaker for gamers. Let me explain.

When I was buying my TV a few years ago, I made sure to check the input lag of all the TVs I considered before making a purchase. I ended up choosing a TV which had a 20ms input lag in game mode, and around 100ms in non-game mode. When I first hooked up my PS4, I immediately noticed the lag in non-game mode. Even things like menus were noticeably laggy. Games were unplayable.

My short, unbiased review of Avatar 2

I went to see Avatar 2 yesterday. It’s over 3 hours long, but I didn’t notice the time passing because it was an entertaining movie with awesome effects. The end.

Why “Never Add to a Losing Position” is the Worst Trading Advice Ever

Disclaimer: Not trading advice.

If you go to YouTube and search for trading videos, you’re sure to find plenty of videos with titles like “Top 5 Rookie Trader Mistakes And How To Avoid Them“, or “WINNERS ADD TO WINNERS! 🏆” One of the most common “rules” internet trading gurus preach is “never add to a losing position”. This will usually be accompanied with similar golden nuggets of advice such as “cut your losses and let your winners run”— which is great advice provided you possess a time machine so you can find out in advance which trades will be your winners and which ones your losers; but why is “never add to a losing position” bad advice? It sounds logical. After all — if you’re in a hole you don’t keep digging, right?

Let’s suppose you’re trading stock X. You believe in stock X long term and believe it will go higher. And you have good reason to believe this. Stock X has been performing very well for the past few months, and its fundamentals are sound.

Let’s suppose you buy 10 shares of stock X at $100 per share. Soon after that, the price of stock X falls to $99. Your trade is now technically “losing”. do you close it?

Maybe stock X is just having a small pullback. Maybe it would be a good idea to add some more here, after all, you’re getting it at a cheaper price, right? Yes. You are right. And this is how professional traders trade. They add to losers. And they think nothing of it because they know that those losing trades are eventually going to be winners.

Suppose stock X dips even further. To $90. Ask yourself the question: “Do I still believe stock X will rise long-term?”. If the answer is yes, then you buy. And congratulations. You’ve just bought stock X at a $10 discount. Suppose you buy 10 more shares at $90. What’s your average entry price?

(($100 × 10) + ($90 × 10) ) / 20 = $95

See that little dashed horizontal line on your chart? That’s your average entry price. Some trading terminals have it and some don’t. For example MetaTrader 4 doesn’t but you should be easily able to find an indicator that calculates and displays it for you. It depends. But if not, it’s easy to calculate it.

If the current ask price is below that line, you buy more. Yes. Even in a losing position. That makes the line go further down. The aim of the game is to get that line as low as possible.

Once the price goes back above it, congratulations! You are now a winner.

Should I Add To Winning Positions?

No. You should never add to a winning position because that will increase your average entry price. If you want to close your trade, that’s fine. Maybe you’ve had a good day and you’re happy with your profits. That’s great. The markets will still be there tomorrow.

What About Doubling (or Tripling) Down?

Never, ever, do this. In fact, when you add to a losing position, never add more than your initial risk. And give yourself plenty of opportunities to catch bigger dips. Increasing your order size as the price drops can get out of hand very quickly. I’ve been there.

Remember, if the price dips 50%, you can buy the same amount of the asset as you did in your original buy for half the price. If the price dips a further 50%, you can buy for a quarter of the original price. Spend the same amount or less buying each dip.

This Seems Like Simple Common Sense, But Why Don’t Traders Do This?

Professionals do. Professional traders do this all the time. It’s known as “Scaling in”. Sometimes you know the market will go up, but you don’t know when, or by how much, or how much it will go down by before it goes back up again. Professional traders don’t know these things either. But they somehow manage to make money.

Professionals also don’t add to winning positions. This seems counter-intuitive, especially to the aspiring retail trader who learned how to trade from YouTube. But adding to a winning position increases the likelihood of it becoming a loser in the future.

I Have a Winning Trade, but What About Dips?

So here’s a situation: you have a winning trade, a nice dip has just happened, and your trade is still in the green (even after dipping) and you want to buy it. Do you buy? Here’s my advice: buy it if you think it’s a good opportunity, but buy it as a new position so you can keep track of it separately, and make sure you use adequate money management.

But remember this: you’ve got a better position open already so if you’re risking more money in this new trade then you’re risking more for less. If you feel you must open a new trade, make it smaller. The goal is for your big trades to have the low entry prices.

Sometimes a trade will go your way right away, and you can soon close it out in a profit, but other times the trade will go against you and you need to “manage” it. Sometimes your best trade is the one that initially goes against you, but then comes good. Have patience. Sometimes there’s nothing you can do except wait.

Trading is boring. If you find yourself feeling excited whilst trading then something is wrong.

In short:

Adding to losing trades is good, provided the market is bullish in the long term.
Adding to winning trades is bad, despite what some guy on YouTube told you.
Use adequate money management.

And don’t risk what you can not afford to lose.

Exploiting Arbitrage Opportunities in Betting Markets

An arbitrage opportunity in financial markets is a time when you can buy an asset on market A for price X and simultaneously sell the same asset on market B for some Y > X and make a guaranteed profit. Arbitrage opportunities generally don’t last long since the buying on market A will cause the price to rise and selling on market B will cause the price to fall until the prices are the same and the opportunity no longer exists.

Such opportunities also exist in betting markets such as PredictIt and Betfair. Today I discovered such an opportunity.

While many believe the 2020 US presidential election is all but settled and Joe Biden is president-elect, he only officially earns that title when the electoral college meet on December 14^th and until then the betting markets are still open and taking bets on the outcome of the election. In betting parlance, the election is still “in play”.

Whatever your political view, one thing is certain: Joe Biden will be the next president or not be the next president. That is certain. The probability of these two outcomes sum to 1.

The screenshot above is from PredictIt, a prediction markets site which specialises in political events. The way PredictIt works is users buy and sell virtual “shares” in a market, with the goal of buying low and selling high, just as in financial markets. Each share has a minimum price of $0.01 and a maximum of $1. When the event is finished and the outcome is known, the market is settled and the price of the shares of the outcome that happened go to $1 and the price of all other outcomes’ shares go to zero.

The higher the price of the shares in a particular outcome, the more probable that outcome. In fact, the price is also the probability. So Joe Biden’s $0.88 price as presidential election winner means he has an 0.88 probability of becoming the next president. Now let’s take a look at Betfair.

Betfair is primarily a sports betting site, but also offer markets on political events including the US presidential election. Betfair has more in common with traditional bookmakers than with PredictIt, but offers a feature many traditional sites do not: Laying. Laying allows a user to play the role of the bookmaker and take the other side of bets made by bettors who believe an event will happen. Laying allows one to bet against an outcome.

Betfair quotes all its market odds using the European style which normalises the stake to 1 and adds on any potential payout to give the odds. The potential winnings on any bet can be calculated by subtracting 1 from the odds and multiplying the result by the stake. For example, Joe Biden’s odds of becoming the next president at time of writing are 1.04, so if I were to bet €100, my payout if he won would be:

(1.04 – 1.0) × €100 = €4

Of course I would get my original €100 stake back too. European odds can be converted into their implied probabilities by simply taking their reciprocal. So Joe Biden’s odds of 1.04 imply a probability of:

1 / 1.04 = 0.9615

Or 96%. Recall the probability we saw on PredictIt for the same event: 0.88. A difference of 8%! So how can we exploit this difference through arbitrage? Take a look at the pink “Lay all” column on the Betfair screenshot. Joe Biden’s lay odds are 1.05. That means if somebody else wants to bet €1 on Biden becoming next president, and I am happy to take the other side of that bet, my liability if Biden wins is that bettor’s potential winnings: (1 – 1.04) = €0.04, plus a small lay commission of €0.01 which I need to pay, giving a total liability of €0.05. If the event does not happen i.e. if Biden does not become president, I get my €0.05 back plus the entire backer’s stake of €1. We can calculate the probability of Biden not becoming next president by taking the probability of him becoming next president and subtracting it from 1:

1 – 0.9615 = 0.0385

Nearly 4%. As a rule of thumb, if you ever have the opportunity to bet on N outcomes of an event where one of the outcomes is guaranteed to happen, and the implied probabilities of the N payouts sum to a number less than 1, you are guaranteed to make money. Any such scenario is an arbitrage opportunity. Let’s analyse our current scenario and check if it’s an arbitrage opportunity. We can bet on Joe Biden becoming the next president with an implied probability of 0.88 on PredictIt and we can also bet on him not becoming the next president on Betfair with an implied probability of 0.0385.

0.88 + 0.0385 = 0.9185

One of these outcomes is guaranteed to happen – he either becomes president or does not become president – and 0.9185 is less than 1 so we do indeed have an arbitrage opportunity. So what if we were to buy one share on PredictIt at $0.88 and make one lay bet on Betfair with a backer’s stake of €1? We part ways with $0.88 to PredictIt and €0.05 to Betfair. Let’s assume $1 is equal to €1 to keep things super simple. After placing our initial share purchase and lay we’re down $0.93. If Biden wins, we lose our $0.05 liability on Betfair but our $0.88 share on PredictIt goes to $1 giving us a profit of $0.07:

$1 – $0.93 = $0.07

On the other hand if Mr. Biden does not win, our PredictIt share drops in value to $0, but we get to keep the backer’s stake on our Betfair lay as well as getting back our risked liability: $1 + $0.05 = $1.05. Total profit in this instance is:

$1.05 – $0.93 = $0.12

So we win 7 cents if Biden becomes president and 12 cents if he does not. This is an asymmetric payoff which may not be desirable, especially if you are somebody with no political knowledge or opinion, who simply wants to exploit the arb opportunity. What if we want to be guaranteed the same profit no matter what the outcome? Let’s have a look at the variables we have:

Bet	Lay
o₁ (Bet odds)	o₂ (Lay odds)
w (Stake)	y (Backer’s stake)
x (Payout)	z (Liability)

Essentially we have two bets on two different markets, with different [European] odds: o₁ and o₂. In the first bet we’re playing the role of the bettor, who stakes some money w, and if successful wins a payout x. Note that payout does not include the stake, which the bettor also gets back if they win. In the second bet we’re playing the role of the bookmaker, taking bets from a bettor who bets y, to whom we pay z if they win. If they do not win, we get to keep y. If we place a bet and a lay, we will either:

Win x, losing z
Win y, losing w

Let’s denote our desired profit to be p. Whatever the outcome, we want to make a profit of p. To achieve this we need to win p plus whatever we lost in the losing bet. So we set the payout and backer’s stake to p + z and p + w respectively.

Bet	Lay
o₁ (Bet odds)	o₂ (Lay odds)
w (Stake)	p + w (Backer’s stake)
p + z (Payout)	z (Liability)

Recall from our discussion on Betfair that the payout is a function of the stake and the odds:

The potential winnings on any bet can be calculated by subtracting 1 from the odds and multiplying the result by the stake.

So payout = (odds – 1) × stake. Turning that into two equations based on the “Bet” and “Lay” columns in the above table we get:

p + z = w(o₁ – 1) [1]

z = (p + w)(o₂ – 1) [2]

Essentially we need two equations which solve for w and z – this will allow us to know how much we need to bet on the two markets. I’ll spare you the messy algebra and just say that the equations we need are:

w = (o₂ × p) / (o₁ – o₂) [1]

z = (p + w)(o₂ – 1) [2]

You may have noticed that the equation for z is unchanged. It can be written in terms of o1, o2 and p but it’s more succinct to solve equation 1 for w and substitute the result into equation 2.

Getting back to our election example, let’s say we want to guarantee ourselves $1,000. We set p = 1,000 then calculate the European odds for the two markets. o₂ is already done because Betfair uses European odds by default, so we have o₂ = 1.05. For PredictIt, to convert the implied probability of 0.88 to European odds we take the reciprocal:

1 / 0.88 = 1.136

We now have our three independent variables:

o₁ = 1.136

o₂ = 1.05

p = 1000

We can substitute these into equations [1] and [2] to get w and z:

w = (1.05 × 1,000) / (1.136 – 1.05) = $12,157.89

z = (1,000 + 12,157.89)(1.05 – 1) = $657.89

How do we know from this how many $0.88 shares to buy on PredictIt? We want our profit to be the same as if it were an exchange such as Betfair – that is: p + z. Our $1,000 target profit (p) plus some more profit to cover for the loss on Betfair (z). If we win, this can be formulated as a function of the buy (b) and sell (s) prices and the number of shares, n:

p + z = (s × n) – (b × n)

Re-arranging to be in terms of n, we get:

n = (p + z) / (s – b)

Plugging in our numbers:

n = ($1,000 + $657.89) / ($1 – $0.88)

n = $1,657.89 / $0.12

n = 13,816

So we need to buy 13,816 shares.

OK, we’re about to start parting with money now, so let’s keep a running total of how much down (or up) we are. First we buy 13,816 Biden shares on PredictIt priced at $0.88 each, costing us:

13,816 × $0.88 = $12,158.08

So our P/L is currently:

-$12,158.08

Next, we stake our $657.89 liability on Betfair at odds of 1.05. So our P/L drops by a further $657.89 making it:

-$12,815.97

Now we wait for an outcome.

Let’s suppose President Biden wins. In this instance we get no money back from Betfair because the bettor who took the other side of bet we laid, won, meaning they get to keep our staked liability of $657.89. However we do have 13,816 shares on PredictIt which are now worth $1 each. If we sell all these shares we make $13,816 it brings our P/L up to:

$1,000.02

In the unlikely event President Biden does not win, then all 13,816 of our PredictIt shares are now worthless but we do get our staked $657.89 Betfair liability back, since the bettor who took the other side of that bet has now lost, and owe them nothing, bringing our P/L up to:

-$12,158.08

This is the cherry. Now for the Sundae: finally, we get to keep the entire backer’s stake, which, at odds of 1.05 is:

$657.89 / (1.05 – 1) = $13,157.89

Bringing our final P/L to:

$999.80

Please note that the math presented here contains some rounding errors. I used about 7 digits of precision in most calculations. The errors in these kinds of calculations can get quite big due to dividing very large numbers by very small numbers. I recommend you use as many decimal places as possible.

Digital Certificates with Alice, Bob and Charlie

I’m currently studying for my CS finals so I’ll make this brief.

Alice likes to buy things from Bob’s website. However there are many impersonators out there pretending to be Bob who are more than happy to take Alice’s money. Alice and Bob both trust Charlie. Charlie doesn’t like the situation Alice and Bob are in so he decides to become a Certificate Authority (CA).

Charlie soon realizes that becoming a CA is actually quite easy and involves typing a few commands into a terminal. Because Charlie is very organized, he begins by creating 3 directories:
charlie$ mkdir keys csrs certs

First Charlie generates a 2048-bit RSA key:
charlie$ openssl genrsa -aes128 -out keys/charlie.key 2048

Next Charlie creates a certificate signing request:
charlie$ openssl req -new -key keys/charlie.key -out csrs/charlie.csr

Charlie then responds to his own certificate signing request by creating and signing his own certificate:
charlie$ openssl x509 -req -days 365 -in csrs/charlie.csr -signkey keys/charlie.key -out certs/charlie.crt

Charlie now has the following files: a key, a certificate signing request (CSR), and a cert which is valid for the next 365 days. Charlie can now delete the CSR if he wants, since he now has his shiny new cert, although he might want to keep it for when he’ll need to create a new certificate in a years time. Charlie is almost in business. There’s just one more step. Since he’ll be making lots of certificates and issuing them to lots of people and legal entities, he’ll need a way of keeping track of the serial number he gives each certificate, to ensure he doesn’t use the same serial number twice.

Since the self-signed certificate Charlie just created is his first, it makes sense that if his next one is his own plus one, they’ll be easy to track, and he’ll have an easy way of counting how many certificates he’s issued. Charlie creates a file called certs/charlie.srl, which contains his own cert’s serial number:
charlie$ echo $(openssl x509 -in certs/charlie.crt -noout -text | egrep Serial -A1 | tail -n1 | sed -e 's/[^a-f0-9]//g') > certs/charlie.srl

This file will be looked after by OpenSSL and incremented every time Charlie creates a new certificate. Charlie is now a CA! Bob now wants to get certified by Charlie. Bob is also very organized and makes some directories to organise his work:
bob$ mkdir keys csrs certs

Bob generates a 2048-bit RSA key:
bob$ openssl genrsa -aes128 -out keys/bob.key 2048

Next Bob creates a CSR:
bob$ openssl req -new -key keys/bob.key -out csrs/bob.csr

Bob sends the CSR to Charlie. Charlie now creates the certificate:
charlie$ openssl x509 -req -in csrs/bob.csr -CA certs/charlie.crt -CAkey keys/charlie.key -out certs/bob.crt

Charlie sends the cert to Bob and Alice can now reliably identify Bob! Next time Alice visits Bob’s website, she downloads his certificate and checks that it is indeed Bob:
alice$ openssl x509 -in certs/bob.crt -noout -text

Bitcoin Difficulty – It’s Actually Quite Easy

I’ll cut to the chase. You’re reading this because you want to understand the concept of bitcoin difficulty but you don’t want a long intro about bitcoin in general, you just want to know how the difficulty algorithm works.

Here’s how:

b = 1 / hp

Where:

b = Block Time

h = Hash rate

p = Probability of finding a block in one hashing attempt

b and h should have the same time units (e.g. seconds) and p, like all probabilities, is between 0 and 1. Having any 2 of these values means the third can be found.

Using this simple equation websites such as blockchain.info can publish the hash rate, currently about 35,000,000 TH/sec (that’s 35,000,000,000,000,000,000).

Blockchain.info don’t find the hash rate by by calling up all the bitcoin miners every month and ask them what their hash rate is, nor is there a centralized mining registry where miners must report their hash rate every month. The hash rate is instead calculated using the above equation, (well actually rearranged a little to be h=1/bp). It’s an extremely cheap calculation. p and b are known. b is the mean time between blocks and that can be calculated by averaging the block time of the last few blocks, and p can be calculated from the difficulty.

But what is the difficulty?

Consider the equation p=x/y. Holding y constant and changing the value of x causes p to change proportionally to x. Difficulty is the inverse of p. d = 1/p if you like. This d is not needed in the calcuulations though. The lower x (and thus p) are, the greater the difficulty. A bitcoin miner is said to “solve” a block if they produce a hash of that block which is lower than the current difficulty, x.

Essentially, mining is randomly guessing numbers from 0 to y until you find one that’s less than or equal to x.

However, this guessing has to be random and done in a very specific way; and if you make a correct guess you have to prove that you made the guess in this way: Computing SHA-256 hashes of a block header.

As a miner, every time you produce a hash you’re essentially generating a random number between 0 and 2²⁵⁶. Because of the way hashing works, you have no control of how these numbers are distributed. You can’t aim at a certain range of numbers, or rule out certain numbers you already tried. All you can do is keep trying again and again, until you get a number less than the current difficulty target. And it’s possible to guess the same number twice. Although hash collisions are highly unlikely.

Difficulty Adjustment

Every 2,016 blocks the bitcoin difficulty is adjusted such that it will make the average block time about ten minutes. If the hash rate is unusually high the average block time will be less than ten minutes and if the hash rate is unusually low it will be more than ten minutes.

Let’s do an example adjustment.

First observe the previous 2,016 blocks and for each one note the time between it and it’s predecessor. Sum up all these observations and divide by 2,016 to find the mean, or use the median — they’re likely to be very similar. We measure this number to be 2m 35s, or 155s. So that’s one variable found: the block time or b.

The other variable is the current p (probability of finding a block), which is burned onto each of the previous 2,016 blocks. The current difficulty is 2²⁴³.

So let’s calculate it:

p = 2²⁴³ / 2²⁵⁶

p = 1 / 2^256-243

p = 1 / 2¹³

p = 0.0001220703125

So that’s the current p. Having b and p we can compute h. Re-arrange the equation:

h = 1 / bp

h = 1 / (0.0001220703125 ✕ 155.0)

h = 1 / 0.0189208984375

h = 52.851612903225806

This is the current hash rate. Using our trusted equation again, this time in terms of p, to compute the probability necessary to make the blocks happen every ten minutes given the current hash rate.

p = 1 / hb

p = 1 / (52.851612903225806 ✕ 600)

p = 0.0000315348307291666

Finally multiply the new p by 2²⁵⁶ to get the newly adjusted difficulty target. Using python:
>>> import math
>>> (2 ** 256) * 0.0000315348307291666
3.65148393387532e+72
>>> math.log(_,2)
241.04730571477836

So the difficulty has got smaller roughly by a factor of 4 — it’s gone from 2²⁴³ to 2²⁴¹. This means there’s a smaller target, thus the probability will be smaller, thus more hashing attempts will fail thus the block time will increase, also by a factor of 4. This makes sense as the original non-adjusted block time was 2m 35s, which is roughly one quarter of 10 minutes.

It’s worth pondering for a second the enormity of these numbers. Searching for a number less than 2²⁴³ in the search space of 2²⁵⁶ is like searching for a needle in a haystack, but 2²⁴³ is a massive number. Far larger than the number of atoms in the earth.

BX Recipes

Today I’m going to talk about bx.

bx (Bitcoin eXplorer) is a little command line tool that allows you to do, erm, bitcoin!, on the command line.

It’s a front-end for the underlying library, libbitcoin on which it’s built. Similar to openssl, bx is a ‘swiss army knife’ type command, offering many subcommands that can do different things. Combining these sub-commands through unix pipes, we can do many practical commands. Combine with bash, and you have even more power. Let’s dive in!

Here’s a bitcoin address:

1LmqvzD5KgNs1a8tcibuiNNM1MfZxLr5pw

As you may or may not know, a bitcoin address is a base58-check encoded representation of a public key hash. It’s generated by SHA256ing the public key, then RIPEMD160ing the result, to give a 160-bit hash, which is known as a “PKH” (Public Key Hash).

To turn this into a standard bitcoin address, a version byte (0x00) is prepended, then a 4-byte checksum (which is another double-SHA256, this time of the version byte concatenated with the RIPEMD160), is appended, giving a total of 165 bytes:

[Version Byte][160bit PubkeyHash][4-byte checksum]

This is then base58-check-encoded to give something which looks like the address above.

Let’s do the reverse. First let’s generate a random bitcoin private key:

[user@bitcoin libbitcoin-explorer]$ bx seed | bx ec-new
69bc69641bd3111e47d55c40b8f1e912577c78edabb1c8ca5d2907bff7a93097

So now have a private key (69bc69641bd3111e47d55c40b8f1e912577c78edabb1c8ca5d2907bff7a93097). Let’s convert that private key into a public key:

[user@bitcoin libbitcoin-explorer]$ echo 69bc69641bd3111e47d55c40b8f1e912577c78edabb1c8ca5d2907bff7a93097 | bx ec-to-public
0380864b7759a43923d2d0c99b142843fc306199a94995e55cb204d9a91fa09feb

This is our public key. There’s only one more step in converting it to a standard mainnet bitcoin address, and that’s the ec-to-address subcommand. But let’s do it 2 ways so we may better understand exactly what ec-to-address is doing. So first the easy way:

[user@bitcoin libbitcoin-explorer]$ echo 0380864b7759a43923d2d0c99b142843fc306199a94995e55cb204d9a91fa09feb | bx ec-to-address
1LmqvzD5KgNs1a8tcibuiNNM1MfZxLr5pw

So there’s our nice short bitcoin address which under no circumstances should you send any money to, unless you want to lose it! Ok now for the long way. First is our hashing step. This involves SHA256ing the public key (bitwise), SHA256ing the SHA256 from the previous step (bitwise), and finally RIPEMD160’ing the SHA256 from the previous step. Bitwise.

[user@bitcoin libbitcoin-explorer]$ echo 0380864b7759a43923d2d0c99b142843fc306199a94995e55cb204d9a91fa09feb | bx sha256
b02260cfb06dfafc55c4c38ace673ce0c902ec0de84fbc75a19adce6bfc9bf9c
[user@bitcoin libbitcoin-explorer]$ echo b02260cfb06dfafc55c4c38ace673ce0c902ec0de84fbc75a19adce6bfc9bf9c | bx ripemd160
d8e5899f2589b7fa9df2513bf2366d2031f2d478

These steps can be combined into one line:

[user@bitcoin libbitcoin-explorer]$ echo 0380864b7759a43923d2d0c99b142843fc306199a94995e55cb204d9a91fa09feb | bx sha256 | bx ripemd160
d8e5899f2589b7fa9df2513bf2366d2031f2d478

Next we prepend our version byte (0x00), giving 21 bytes:

00d8e5899f2589b7fa9df2513bf2366d2031f2d478

Then we double-SHA256 it:

[user@bitcoin libbitcoin-explorer]$ echo 00d8e5899f2589b7fa9df2513bf2366d2031f2d478 | bx sha256 | bx sha256
08c131a41c9ba6763e87c037f3eb9fa75b8f593750e8d070cd03cff71c07f02d

Next we take the first 4 bytes from this result (0x08c131a4), and append that to the [version byte|hash160] string, giving 25 bytes:

00d8e5899f2589b7fa9df2513bf2366d2031f2d47808c131a4

Finally, let’s base58-encode this and see what we get:

[user@bitcoin libbitcoin-explorer]$ echo 00d8e5899f2589b7fa9df2513bf2366d2031f2d47808c131a4 | bx base58-encode
1LmqvzD5KgNs1a8tcibuiNNM1MfZxLr5pw

Same address! Done the long way.

Setting up a secure HTTP server on a FreeBSD virtual machine

NB: Before you begin following this article you’ll need to understand how to use vi, as we’ll be working at a command-line interface with this as the only text editor at our disposal. If you don’t know how to do basic text editing in vi I suggest you read a few articles and familiarize yourself with it otherwise you won’t have a good time Also if you’re on an IPv6 network, you can follow this article if you want but it’s all inIPv4 so..you’ve been warned

Today I’ll take you through the set-up of a secure (HTTPS) server on a FreeBSD virtual machine which you can use to host your own website for dev purposes, or for educational purposes, or whatever.

We’re going to be using VirtualBox, which can be downloaded here and FreeBSD 10.2 which is the latest stable release at time of writing, although you probably won’t have any problems using a newer release.

The web server we’re going to use is NginX, which is a highly configurable, simple, fast, secure and powerful web server.

What are the advantages of using a virtual machine?

Glad you asked that question. I’ll start with the disadvantages: A virtual machine runs software slightly slower and consumes some more resources than the host machine it’s running on. The advantages are…everything else.

What are the advantages of using a FreeBSD?

FreeBSD is a industrial-grade operating system that is extremely secure, fast, and surprisingly simple to use. Ever used OSX? You’re using FreeBSD. Yeah. OSX is basically FreeBSD with lots of `skinning’. Ever played a PS4? It’s operating system is a modified FreeBSD, same with the PS3. It’s also used for mission-critical high-traffic networking equipment such as load balancers, firewalls, email servers and webservers, basically any really important application where Linux is just not “up to the job”.

And speaking of webservers, that’s what what we’re going to be doing.

If you haven’t already done it, install VirtualBox on your Mac, Windows or Linux machine.

Next you’ll need to head over to FreeBSD.org and grab the 64-bit ISO disk image of the 10.2 release, Click Production 10.2, scroll down to ftp, click on the first link, which is this, then click on the 64-bit CD image, which is this, and let it start downloading.

Go back into VirtualBox, and click the New icon to start creating your virtual machine. When you create something, a lot of times the first thing you do is give it a name, and creating a VirtaulBox virt is no exception. In this case I’ve gone for `BSDBox’, but you’re free to use anything you like, as long as it doesn’t contain any illegal characters so go on, pick something wacky, type in your crazy name then select `BSD’ under type and `FreeBSD (64-bit)’ under version.

Next you choose how much of your computer’s memory will be allocated for this virt, in this case I’ve gone for 512MB which I’m hoping will be enough to run NginX and get me through this article. Feel free to use more if you can afford it, but not less, please.

Next we create a virtual Hard Drive… do what the picture says.

Do what the picture says.

Again.

Wait, might take a while…

Ok now we’ve got a virt set up with an 8GB harddrive and 512 MB of memory but no operating system.

If you click `Start’ you’ll get something like this. FYI, when you do this VirtualBox might ask you if you want mouse pointer integration which is supported on most modern operating systems (ie. operating systems that are capable of knowing that they’re running on a virtual machine), and the best choice is to pick yes. This `traps’ the mouse pointer inside the VirtualBox window which just makes it feel more like a real computer. The mouse pointer can be released by pressing the right ctrl key.

Installing FreeBSD

Next we insert our virtual CD into our virtual CD drive and install the operating system. Open up the settings window for your virt, click `Storage’, highlight the DVD drive under `Storage Type’, click the picture of a DVD on the far right, click `Choose a virtual CD or DVD file…’, browse to the one you downloaded from FreeBSD.org, or if it’s not downloaded yet, wait.

BSDBox - Settings_019

Click Ok, then click Start to start up your virt. Hit enter when you see the black screen

Hit Enter to install

Choose a keymap, test it, then choose the first option to Continue

Type in the same name you chose for your virt as the hostname, hit Enter

For speed just choose 32-bit compatibility libraries and ports tree, hit Enter

Choose guided disk setup.

Entire disk.

GUID Partition Table.

Enter

Enter.

Wait.

Choose a root password. BSD isn’t very picky here, in fact I think you may be able to press Enter here and leave it without a root password so type carefully.

The next screen you see will be asking you about configuring network interfaces. Just press Esc to take you to a main menu then press enter to exit the installer.

After a few seconds it will ask you if you want to make any final changes, choose `No’, wait for a few seconds and the virt should shut down. If not, choose “ACPI shutdown” from the Machine menu. Wait for the machine to go into the stopped state. Now remove the virtual disk from the virtual drive otherwise when you switch it on it will run the installer again.

Networking

At this point we have got a fresh install of FreeBSD on our virtual hard drive and the next step is to configure VirtualBox such that the FreeBSD virt has it’s own IP address on the same network the host machine is on. This means we can open a web browser and browse to a page that’s hosted on the virt from our host machine, or from any machine on the network. VirtualBox has lots of options for configuring networking and the default one is `NAT’. Click Settings then click Network to bring up the network settings.

As you can see `NAT’ is selected which means VirtualBox will run a sort of “virtual router” which the guest OS (FreeBSD) thinks is a real router, so when you start up a virt, the guest OS sends a DHCP request over it’s virtual network card, which is picked up by the virtual router which responds by giving it an IP address, subnet mask, gateway address and DNS address(es). Next, when a packet is sent from the virt to the virtual router, the virtual router changes the source address of the packet (and possibly the port) to be the address of the host machine then forwards it off to the real router to get to it’s final destination. When a response comes back VirtualBox again translates the destination address (and possibly the port) of the packet to match the address and port from which the first packet was sent.

This is convenient and fairly secure for tasks such as web browsing and email but not if we want to run a server since we want others on the network to be able to contact us directly, this is achieved by something called a `Bridged Adapter’, so change from NAT to bridged adapter and click OK.

Start up the virtual machine again. A bridged adapter sends the packets emitted from the virt directly to the host machine’s network card, circumventing the host’s TCP/IP stack. The network card handles the Ethernet protocol only and has no clue about what is inside any of the packets it’s transmitting and receiving so VirtualBox can just inject packets generated by the virt so it’s mixed in with the traffic from the host machine, the network card forwards them to the nearest router, the router may or may not notice that there are now 2 IP addresses associated with the same MAC address, and almost certainly won’t care, and will forward them on and deliver back the responses.

When the responses come back in, VirtualBox will be sniffing all the packets and will grab any that have the virt’s IP address as a destination. At the same time the host machine’s TCP/IP stack will ignore any packets that don’t have it’s IP address.

When you see the login screen type `root’ as the login and the password you chose earlier. Once you’re logged in type

# ping www.google.com

You should get a message indicating that the host couldn’t be found. That’s to be expected. What we’re going to do now is enable DHCP on the virt and reboot the machine, when it reboots it should get an IP address (and the rest) from the host machine’s DHCP server, which is probably also the gateway. Once we get this IP address and test our network connection and if it works, (ie. we can ping google.com), we’ll make a note of the IP address/netmask/gateway, etc then set them manually so the virt will always use these settings. This means we don’t have to go to the bother of checking what kind of network we’re on, although chances are you’re just on a standard 192.168.x.y, some people have weird network configurations, but more importantly we’ll have a static IP address which will persist across reboots which is essential for running our webserver.

Type the following command:

# echo 'ifconfig_em0="DHCP"' >> /etc/rc.conf

Then this:

# reboot now

This adds a command at startup which configures the network interface (em0) to get it’s address via DHCP. Wait for the virt to reboot, log in again and try pinging google.com. You should get replies. Press Ctrl-C to break out of ping. Now we need to determine the virt’s IP address, netmask, gateway and DNS servers. Type ifconfig and you should see something like this:

Make a note of the inet and netmask in the em0 section, in my case they are 192.168.1.16 and 0xffffff00. We’ll have to convert the netmask to dotted decimal notation, this can be done by stripping off the 0x then grouping the eight remaining digits into groups of two to give ff.ff.ff.00, finally use a hex calculator to convert each of these bytes into decimal, to give 255.255.255.0

That’s the IP and netmask sorted. To get the gateway address type the command

# netstat -r -f inet

which will display the local routing table. There should be 4 columns, the first tow of which are Destination and Gateway. Look under Destination and look for default, check what is listed in the corresponding Gateway column, and… that’s your gateway address!

Finally you need your DNS address(es). Type

# cat /etc/resolv.conf

You should see something like

nameserver X.X.X.X
...

There’s a high probability that there’s only one nameserver listed and it has the same address as your gateway. Whether this is true or not, it doesn’t matter, because the good news is you can just leave this file alone and it will continue to be used as the nameservers. Next you’ll want to open the file /etc/rc.conf in vi

# vi /etc/rc.conf

Remove the last line and replace it with

ifconfig_em0="inet X.X.X.X netmask Y.Y.Y.Y"
defaultrouter="Z.Z.Z.Z"

of course replacing X.X.X…. with your IP address, netmask and gateway respectively. Save and close the file and reboot your virt again. When you log in, do another ping test to check that you’re online, if you get replies, fantastic. While you’re at it, open a command prompt on your host OS and type

$ ping bsdbox

Or whatever name you chose for your virt. You should also get replies. As a final step, we’re going to edit our hosts file so it has the hostname-to-IP mapping. Sendmail was giving me problems on my virt and it probably will on yours too if you don’t take this step, so to do it all in one go type (CAREFULLY):

# echo 192.168.1.3 bsdbox >> /etc/hosts

inserting your own IP address and hostname instead. Restart the virt again. It would be no harm to do this on your host machine and any other machine(s) you plan on accessing the webserver from. It’s done the same way in Linux (as root), for OSX and Windows google “editing hosts file in (OSX/Windows)” and you should find tons of info on how to do it.

Installing NginX

Now it’s time to install our web server. First you’ll need to enter this command to get FreeBSD to update it’s ports tree

# portsnap update

This will take a few minutes, next you’ll need

# portsnap extract

This will take even more time. What’s happening here is a directory tree, rooted at /usr/ports is being built for the entire FreeBSD ports collection. That is: software that’s been ported to FreeBSD and has been tested and is known to work on FreeBSD. The ports collection is divided into categories for different types of software: such as development, games, web, math, multimedia, and so on.

When that’s done, and it will take a while, you’ll want to navigate to the /usr/ports/www/nginx directory. Now we’re going to compile NginX. If you type lsyou’ll notice a Makefile in there, so type

# make install clean

to begin building NginX. After a while you should see a blue screen asking what features of NginX you want, Just make sure HTTP_SSL is selected, then hit Enter to continue the install.

When all that is finished, NginX should be installed on your virt. Before we reboot again, we’re going to set NginX to start up automatically on boot, that means adding one line to rc.conf:

# echo nginx_enable="YES" >> /etc/rc.conf

reboot the virt again and wait till you see the login screen. By this time NginX should have started a basic HTTP server listening on port 80. Open a web browser and into the address bar type http://<hostname>, using your hostname of course. You should see something like this:

So we’ve got a webserver up and running serving normal unencrypted HTTP traffic. Getting there..

Generating SSL Certificate and Key

Navigate to the /etc/ssl directory of your BSD system. Issue the following commands to create two directories:

# mkdir keys
# mkdir certs

Now we’ll create a private key, remember to substitute the name of your host in place of `bsdbox’:

# openssl genrsa -aes128 -out keys/bsdbox.key 2048

This generates an RSA private key with a 2048-bit modulus, which is considered secure at time of writing. You’ll also be asked for a passphrase for your key. Choose a strong password and make a note of it. Next we’ll secure this file by making it unreadable to anybody but root:

# chmod 400 keys/bsdbox.key

Next we’ll make a certificate signing request:

# openssl req -new -key keys/bsdbox.key -out bsdbox.csr

You’ll be asked for some information on your `Company’, fill in the details; for the FQDN use your hostname, and for email address use root@<your hostname>. Finally we’ll create and self-sign the certificate which contains our public key and is valid for the next 365 days:

# openssl x509 -req -days 365 -in bsdbox.csr -signkey keys/bsdbox.key -out certs/bsdbox.crt

You can delete bsdbox.csr file now as it’s no longer needed.

Configuring NginX

We’re nearly there. Open up /usr/local/etc/nginx/nginx.conf in vi. This is the main config file for NginX and is hierarchical in structure. Observe that there’s a http { ...} section which encompasses all the settings and inside that there’s a server{ ... } section which holds all the settings for the default website on our server. We’re only going to be editing what’s inside the server{ ... } section.

Firstly you’ll see

   listen 80;

Change this to

   listen 443 ssl;

Next change server_name from localhost to <yourhostname>; Don’t forget the semicolon!, (you need a semicolon after everything in NginX configs) then under that add the following 3 lines:

   ssl on;
   ssl_certificate /etc/ssl/certs/&lt;yourhostname&gt;.crt;
   ssl_certificate /etc/ssl/keys/&lt;yourhostname&gt;.key;

Here’s a picture of my vi screen afer making these changes:

These are the minimal changes you need to make to have a HTTPS server up-and-running. Save the file and exit. You’ll now need to restart NginX, you can either use the command

# nginx -s reload

or if you’re like me you can just reboot the virt. Whichever you do, at some stage you’ll be asked to provide the passphrase for your private key so NginX can use it. Type it in, and wait for the server to start.

Testing Your Server

Open a browser window and enter the address https://<yourhostname>, you should see a warning telling you it’s unsafe to proceed, or that this connection is untrusted.

Follow these instructions to view the page

Firefox: “I understand the risks”, “Add exception”, “Confirm security exception”

Chrome: “Advanced” , “Proceed to…”

And there it is in all it’s glory! A very secure webserver using 2048-bit RSA encryption hosted inside a virtual FreeBSD instance. Of course to keep it secure you’ll need to log out of FreeBSD every time you leave the computer alone since there’s only one root user, but ignoring the “physical” risks, this server is pretty much unhackable. An output from nmap on my Linux host machine:

root@CodeCook:~/ca# nmap bsdbox

Starting Nmap 6.40 ( http://nmap.org ) at 2015-10-21 08:44 IST
Nmap scan report for bsdbox (192.168.1.3)
Host is up (0.00059s latency).
rDNS record for 192.168.1.3: BSDBox
Not shown: 999 closed ports
PORT STATE SERVICE
443/tcp open https
MAC Address: 08:00:27:52:60:43 (Cadmus Computer Systems)

Nmap done: 1 IP address (1 host up) scanned in 44.05 seconds

As you can see the only open port is 443 because that’s the only software we installed. FreeBSD assumes nothing about what you’re going to do with it and errs on the side of caution. It’s a generic operating system just built to work and nothing else.

I’ll follow up this article with one where we will create our own certificate authority and install a Root cert on the client browser(s) so we can access our secure server without all the annoying warnings.

Perl: references, scalars, etc

A lot of people are put off using Perl because of what they believe is over-complicated syntax when dealing with references. I’m on a mission to dispel this rumour and give a simple guide to Perl references, the best ways to think about them and draw some comparisons with C pointers, and hopefully get more people using Perl.

The Perl language may not be as popular as it used to be but there’s still tens of millions of lines of Perl code out there that needs people to maintain it.

We’ll start off with the humble scalar: A scalar is a variable with a name starting with $, so if we wanted to define a scalar called $s, we’d write:

my $s;

A scalar can be any of 4 things:undefined, a number, a string, or a reference. I’ll repeat that. A scalar (that is: a Perl variable that begins with a $) can be either:

1. Undefined, as in the example above; we’ve declared a varaible $s but not said $s=anything
2. A number, for example:

my $s=42;

3. A string, for example:

my $s="hello";

4. A reference, for example:

my $s=\$y;

See the back-slash? That’s the same as the & operator in C, it gives the address of $y and stores it in $s. $s now contains the address of $y. Let’s write a simple program to demonstrate this.

#!/usr/bin/perl

use strict;
use warnings;

my $y="Noel";
my $s=\$y;

print $s ,"\n";
print $$s ,"\n";

The output from this program:

SCALAR(0x1f43310)
Noel

2 lines of output: the first is SCALAR(0x1f43310) which is what we get when we try to print $s directly. Not very useful; it tells us $s is a reference to a scalar, which itself may be one of the 4 kinds of scalar listed above. But if we put another $ in front of $s we get the scalar that $s points to, which is the string “Noel” (AKA $y). This $ put in front of the scalar is a scalar dereference operator and is similar to the * operator in C. This is kind of confusing because in C the * symbol has 3 uses: Multiplication, Pointer definition and Pointer dereference. Similarly in Perl the $ has two uses: 1) Denoting a scalar, and 2) dereferencing a scalar that references ANOTHER SCALAR.

Arrays

Moving on, Arrays (or lists) are the second kind of data structure we encounter in Perl, and an array is defined with a @. So…

Scalar … $
Array … @

If it starts with a $ it’s a scalar, and if it starts with a @ it’s an array. Got it? Good. Let’s define an array and print it’s contents.

#!/usr/bin/perl

use strict;
use warnings;

my @beatles=("John","Paul","George","Ringo");
print "@beatles\n";

The output from this program:

John Paul George Ringo

This program declares and initilizes an array called @beatles in one line, then prints the array. At this point you should be wondering “Can I use the \ operator to access the memory address of @beatles?” and the answer is yes. Let’s add a few lines to the program:

#!/usr/bin/perl

use strict;
use warnings;

my @beatles=("John","Paul","George","Ringo");
print @beatles , "\n";

my $beatles_ref=\@beatles;
print $beatles_ref , "\n";

Ok so we’ve created a scalar called $beatles_ref which holds the address of @beatles. But if we try to print $beatles_ref, we get

ARRAY(0xf96360)

This tells us $beatles_ref is a reference to an array. So how do we dereference it? I’ll tell you how. Using the array dereference operator, that is: @{}

#!/usr/bin/perl

use strict;
use warnings;

my @beatles=("John","Paul","George","Ringo");

my $beatles_ref=\@beatles;
my $beatles_ref_ref=\$beatles_ref;

print @beatles , "\n";
print @{$beatles_ref} , "\n";
print @{$$beatles_ref_ref} , "\n";

I’ve jumped the gun a little with this program but study it, and run it, and I’m sure you’ll figure it out. This program should print:

JohnPaulGeorgeRingo
JohnPaulGeorgeRingo
JohnPaulGeorgeRingo

What we’ve done here is made an array called @beatles with 4 elements, then made a scalar called $beatles_ref, set it to point to @beatles, then we created another scalar called $beatles_ref_ref and pointed that to $beatles_ref. We then printed the @beatles array 3 times, first by using the @beatles array variable, then by using the array dereference operator @{} on $beatles_ref, and finally by double-dereferencing the $beatles_ref_ref variable to access $beatles_ref, then $beatles. Here’s the same program in C++11:

#include <stdio.h>;

int main(){

   char* beatles [4] ={(char*)"John", (char*)"Paul", (char*)"George",(char*)"Ringo"};
   char** beatles_ref = (char**) &beatles;
   char*** beatles_ref_ref=&beatles_ref;

   for (char* &beatle:beatles){
      printf("%s",beatle);
   }
   printf("\n");

   // we must figure out the length of the array now since we cannot use
   // C++11-style for loops on a pointer
   int arrlen=sizeof(beatles) / sizeof(char*);

   for (int i=0; i<arrlen; i++){
      printf("%s",beatles_ref[i]);
   }
   printf("\n");

   for (int i=0; i<arrlen; i++){
      printf("%s",(*beatles_ref_ref)[i]);
   }
   printf("\n");

   return 0;
}

My eyes are bleeding. As you can see we can only use the concise C++11-style for loop on the ‘beatles’ array since this is a static chunk of data and it’s size is hard-coded into the program, if all we have is a pointer to it we need to manually compute or otherwise determine the length of the array. All of this is automagic in Perl, so, eh..use Perl. Let’s get on to hashes.

Hashes

Here’s a Perl program that defines a hash called %beer and assigns it two attributes: Name and Alcohol, then prints them.

#!/usr/bin/perl

use strict;
use warnings;

my %beer=(Name=>'Heineken', Alcohol=>0.05);

print $beer{Name} , "\n";
print $beer{Alcohol}*100 , "%\n";

So as you can see we use the % character before the variable name to declare a hash. So, to recap:

Scalar … $
Array … @
Hash … %

%beer is a hash, not a hash reference, although to access the attributes of %beer, we use the notation $beer{…}, which admittedly is confusing, but is less confusing if you think of the $ as referring to the scalar attribute rather than the containing hash. So you have

$<hashname>{<attributename>}

to access a scalar element called in the hash .

Like arrays we can also create a reference to a hash:

my $beer_ref=\%beer;

and we access the hash’s attributes with the -> operator like so:

print $beer_ref->{Name} , "\n";

$beer_ref is colloquially known as a hashref, (a reference to a hash), which crop up a lot in Perl, and, like arrays, Perl knows $beer_ref is a hashref because if we do

print $beer_ref;

we get:

HASH(0x15c5488)

But wait! we can create a hash another way! And this is the way I nearly always create hashes… We can declare a scalar and point it to an anonymous hash, like so:

my $hashref={ Name'Peter', Age='45' };

We’ve skipped the creation of a hash variable and gone straight to assigning a hash to a hashref. Did I mention we can also do this with arrays? I don’t think I did…

my $arrayref=["John","Paul","George","Ringo"];
my $hashref={Name=>'Peter', Age=>'45'};

See? Square brackets for the anonymous array and curly brackets for the anonymous hash and we can dereference them in a similar way to arrayrefs, using the %{} operator. So a %{$hashref_to_foo} will be treated the same as a %foo.

For completion:

#!/usr/bin/perl

use strict;
use warnings;

my $arrayref=["john","paul","george","ringo"];
my $hashref={Name=>'peter', Age=>'25'};

my @array=("JOHN","PAUL","GEORGE","RINGO");
my %hash=(Name=>'PETER', Age=>'45');

# Printing the arrays
print "@{$arrayref}\n";
print "@array\n";

# Printing the hashes
print "$_:$hash{$_} " for (keys %hash); print "\n";
print "$_:$hashref->{$_} " for (keys %{$hashref}); print "\n";

# Let's now re-assign $arrayref and $hashref to point to $array
# and $hash respectively and print them again

$arrayref=\@array;
$hashref=\%hash;

print "@{$arrayref}\n";
print "$_:$hashref->{$_} " for (keys %{$hashref}); print "\n";

# At this point the original values of $hashref and $arrayref
# are lost forever 😦

That code sums everything you’ve learned so far. So round brackets for the direct array and hash structures; Square brackets for anonymous arrays and curly brackets for anonymous hashes. But we prefer using references, because a reference is a scalar and a scalar is small, and small things can be moved around faster and stored using less space, but more importantly if you’re using functions (subs in Perl), using references is essential.

Subs

Compared to other languages, writing functions is syntactically very simple in Perl. There’s no need to specify a return type or function parameter list. Every function (aka sub) has an implicit parameter list (aka array) called @_.

So if I declare a function called “foo” which prints the contents of an array I could write:

sub foo{
   print "@_\n";
}

and that should do it. Should I want to call foo, I could write

foo;

which calls the foo function, with an empty parameter list, which will just print a blank line. If I wanted to pass it our @array array from the last listing, I’d say

foo @array;

which would give me

JOHN PAUL GEORGE RINGO

which prints @array (known as @_ in foo). BUT… If I write

foo @array,"STUART";

I’ll get:

JOHN PAUL GEORGE RINGO STUART

And this is a problem. What Perl does here is when foo is being called it goes through the parameter list and when it finds a scalar it adds that scalar to the @_ list, but when it finds an array it adds each element of that array to @_, making the elements of @array indistinguishable from any arbitrary scalars (ie “STUART”) in the parameter list, so foo doesn’t know which elements are from @array. This is useful in cases where foo just wants to do a simple job on each element of the array (such as run a regex) and you might have several arrays and several scalars on which you want to perform this operation; you can simply do something like this:

#!/usr/bin/perl

use strict;
use warnings;

sub capitalize{
   map{ $_ = uc $_ } @_;
}

my @beatles=("john","paul","george","ringo");
my @stones=("mick","keith","ronnie","charlie");

my $fifth_beatle="stuart";
my $stones_manager="andrew";

print "Beatles: @beatles\n";
print "Fifth Beatle: $fifth_beatle\n";
print "Stones: @stones\n";
print "Stones Manager: $stones_manager\n";
print "\n";

capitalize $stones_manager, @beatles, $fifth_beatle, @stones;

print "Beatles: @beatles\n";
print "Fifth Beatle: $fifth_beatle\n";
print "Stones: @stones\n";
print "Stones Manager: $stones_manager\n";
print "\n";

Here we’ve defined two arrays: @beatles and @stones, two scalars: $fifth_beatle and $stones_manager, printed them, and then decided we want to capitalize them all in one fell swoop. So we pass all four of these in no particular order to out capilalize function which builds the @_ array (“andrew”,”john”,”paul”,”george”,”ringo”,”stuart”,”mick”,”keith”,”ronnie”,”charlie”), then runs a map function on the array which runs a lambda function ($_ = uc $_ ) which reassigns the array element ($_ to an upper-case version of it (uc $_)). Doing this in Java would take lots of CPU cycles and coding time doing array manipulation and typing parameter lists. Furthermore if we want a copy of the newly capitalized and concatenated array we just need to change one line:

my @all=capitalize $stones_manager, @beatles, $fifth_beatle, @stones;

And we have an array with them all capitalized then taped together in the order they were sent in, nice and concise.

Passing Arrays and Hashes to Subs

Perl has a dirty little secret and it’s that you can pass a hash to a subroutine then `cast’ @_ into a %hash inside the subroutine which works but I’m sure you’ll agree is illogical and horrible.

#!/usr/bin/perl

use strict;
use warnings;

sub foo{
   my %hash=@_;
   print $hash{bar};
}

my %hash=(bar=>'baz');

foo(%hash);

The `nice’ way to do it is to pass all arrays and hashes to functions as references. So the above program would become

#!/usr/bin/perl

use strict;
use warnings;

sub foo{
   my $hashref=shift;
   print $hashref->{bar};
}

my %hash=(bar=>'baz');

foo(\%hash);

It’s also faster (I think). FYI, the shift function removes the first value off the array @_, which is our hashref. I could have equivalently used

my $hashref=@_[0]

Other Neat Things We Can Do

To create a two-dimensional array we simply fill an array with references to other arrays:

#!/usr/bin/perl

use strict;
use warnings;

sub print_2d_array{
   my $arr_ref=shift;

   foreach my $ele (@{$arr_ref}){
      print "@{$ele}\n";
   }
}

my @two_d_array=(
   ['A','B','C','D','E'],
   ['F','G','H','I','J']
);

print_2d_array \@two_d_array;

If we have a function that returns an array (such as sort or map), we can enclose the function call in [ and ] to give us a reference (similar to how \ converts an array variable to a reference):

#!/usr/bin/perl

use strict;
use warnings;

sub three_random_numbers{
   (rand,rand,rand);
}

my $arr_ref=[three_random_numbers];

print "@{$arr_ref}\n";

# or even (!)
print "@{[three_random_numbers]}\n";

Reversing this process, we can also obtain an array from a function that returns an array reference:

#!/usr/bin/perl

use strict;
use warnings;

sub three_random_numbers{
   [rand,rand,rand];
}

my @arr=@{(three_random_numbers)};

print "@arr\n";

# or even (!)
print "@{(three_random_numbers)}\n";

Similarly, with hashes, we can use { and } to convert the return value of random_person to a hashref

#!/usr/bin/perl

use strict;
use warnings;

sub random_person{
   (
      Name=>('Randy','Donna','Betty','Harry')[int(rand()*4)],
      Age=> int(rand()*100)
   );
}

my $hashref={random_person};

print "Name=$hashref->{Name} Age=$hashref->{Age}\n";

And for completion’s sake:

#!/usr/bin/perl

use strict;
use warnings;

sub random_person{
   {
      Name=>@{['Randy','Donna','Betty','Harry']}[int(rand()*4)],
      Age=> int(rand()*100)
   };
}

my %hash=%{(random_person)};

print "Name=$hash{Name} Age=$hash{Age}\n";

Notice the way we have to enclose the call to random_person in ( and ) ? This ensures we are dereferencing the return value
of random_person, not random_person itself, which is a subroutine, and did I mention that this can also be referenced using
\&random_person ? Yeah, you can do that too. More on that in my next Perl post.

Developing an old-school 2D mobile-friendly loop-based platform game using canvas and touch API (PART 1)

Back in my youth I used to write my own games. They weren’t very good games, but they were just about playable and thought me many concepts about coding. For this article I originally wanted to target the game to 3 platforms: Computers with keyboards, Smartphones and tablets, and The Sony PS4, but after doing some checking it seems the PS4’s gamepad isn’t accessible from Javascript (thanks Sony), so I’ll concentrate on the first two.

I want to write something that closely represents Super Mario Brothers, without infringing on copyright. This was originally released on the Nintendo NES way back in 85. The NES came with a control pad with two buttons: A and B. and a 4-button D-pad, giving a total of 6 buttons. If you were rich, you could buy the 2-controller pack, which meant 2 people could play the game by taking turns but they needed different controllers. (Yes, it makes no sense now either)… maybe they were more worried about germs back in the 80s..who knows.

I’m going to go slowly with this tutorial cause I want to develop the framework first before we delve into the game proper. That means handling various screen resolutions, checking for support of APIs, and abstracting this stuff away as much as possible from our game code. (Trust me it will be better in the long run). Also, you may want to be aware that I’m a fullscreen kind of guy and I’ll be coding this game such that the game takes up as much of the screen as possible. This is essential on smartphones where the display is small but also essential on web browsers where the user’s attention could otherwise be stolen by ads and other such distractions.

I’ll begin with the template I’m using nowadays for most of my web work, plus a few extra bits.

<!DOCTYPE html>
<html>
<head>
<meta charset="utf-8">
<meta name="viewport" content="width=device-width, initial-scale=1.0, maximum-scale=1.0, user-scalable=0">
</head>
<body></body>
</html>

Nothing too complex there, you may not be familiar with the “viewport” meta tag. This is a mobile-specific tag that needs to be added to stop the user from zooming in/out the webpage and also allows the developer to hard-code a width (in pixels) so mobile browsers with different resolutions will automagically scale the content up/down so the page appears with the same proportions no matter what screen resolution their device has. We’re not going to use this feature so we’ve set it to “device-width”. (my reason is because I suspect this will add an extra layer of processing on the device which will put more pressure on the CPU and give less cycles to our game which we want to run as smoothly as possible).

Abstracting the gamepads

We intend our platform game to run on touchscreen devices as well as “traditional” computers, but there’s a third type of device we’d like to support and that’s computers with gamepads attached. There’s a relatively young API that modern browsers are starting to support and that’s the Gamepad API. It allows gamepad events to be heard by javascript thus allowing a HTML-based game to be controlled using a gamepad. I’ll get on to that later.

For now let’s forget about how we’re going to read the physical hardware or even what kind of hardware it is (if it is hardware), and write some simple javascript which will define our 2 Nintendo NES joypads (gamepads):

var gamepads=[
   {up:false,down:false,left:false,right:false,A:false,B:false}, // Player 1
   {up:false,down:false,left:false,right:false,A:false,B:false} // Player 2
];

We can think of this structure as the ‘middle man’ between our (messy) API code and our (nice) game code. The game programmer will never ‘see past’ this structure and will simply read the boolean values to see whether a particular button is or isn’t pressed.

What did I mean by loop-based?

Most web programming these days is event-based, but all the cartridge-based games back in the 80s were loop-based. The game centered around a “main loop” which would run 25 times a second (or 50) typically doing the following tasks:

while (alive){
   updatePlayer();
   updateBaddies();
   detectCollisions();
   drawBackground();
   drawSprites();
   waitVSync();
   flipBuffers();
}

Needless to say this is a very rough description. updatePlayer() would update the player’s sprite (let’s call him Mario), by reading the gamepad’s buttons and reading some other variables related to the player such as current X/Y coordinates, X/Y velocity, whether he was big mario or small mario / whether he was “dying”… and based on these it would arrive at the correct X/Y coordinates and the correct mario sprite to render. updateBaddies() would do something similar for all the baddies on the screen. detectCollisions() would check if Mario has come into contact with any baddies, drawBackground() and drawSprites() would draw the background/ sprites in their correct places and waitVSync() would wait until the TV’s beam had reached the bottom of the screen and returned back to the top of the screen ready to begin the next frame. This ensured a steady frame rate as long as all of the work was done before that beam got to the bottom of the screen. If a game missed that deadline, it would have to wait for the next VSync thus causing that frame to stay onscreen for twice as long as it should have.

There was also another colsely-related concept called double buffering that all games needed to implement in order to look “good”. I have a suspicion that we shouldn’t need it for our canvas-based game. I wrote one a few years ago when canvas had just been invented and I did need to implement a double buffer, which meant 2 canvases (canvii?) stacked on top of each other and while one was being drawn on the other was on display to the viewer, then once the first one is fully drawn it’s put on display and the other one is flipped. This involved changing the z-Index values and some messy css positioning code. I suspect this problem has been sorted out in newer browsers and the programmers have devised some clever way of rendering things to the canvas in “bunches” then displaying it all in an instant, overwriting what was previously there. This would make sense since it ensures less resources to be consumed by the browser and makes coding easier for the developer (us). I am just speculating here though and I could be wrong.

Our main loop

Without further ado, here’s our main loop:

var mainLoop=setInterval(function(){
// Loop code goes here
},20);

This function will run every 20ms , or every .02 of a second, giving, (1/.02)=50 loops per second. or 50Hz. It would be nice to have it run at 100Hz but this might put too much strain on mobile devices. It may not be smooth enough for the young gamers of today who demand ultra high frame rates. So if they want 100Hz they are welcome to modify the javascript and play at 2X the speed.

Getting the canvas out of the way

I hate CSS. I’d like to keep the CSS to an absolute minimum for this tutorial so let’s try and nail the canvas code and get it positioned properly before we move on to the good stuff.

game.html:

<!DOCTYPE html>
<html>
<head>
<meta charset="utf-8">
<meta name="viewport" content="width=device-width, initial-scale=1.0, maximum-scale=1.0, user-scalable=0">
<script type="text/javascript" src="game.js"></script>
<style>
body{
        margin:0px;
        padding:0px;
        overflow:hidden;
}
#theCanvas{
        margin:0px;
        padding:0px;
        border:0px;
        width:100%;
        height:100%;
}
</style>
</head>
<body onload="game.init()" onresize="game.resizeCanvas()">
<canvas id="theCanvas"></canvas>
</body>
</html>

game.js:

var game=(function(){
	var cvs; // reference to the canvas itself
	var ctx; // canvas's 2D drawing context
	var frame=0; // counts the number of frames in our game
	
	var gamepads=[
		{up:false,down:false,left:false,right:false,A:false,B:false}, // Player 1
		{up:false,down:false,left:false,right:false,A:false,B:false} // Player 2
	];

	var resizeCanvas=function(){
		cvs.style.width=window.innerWidth+"px";
		cvs.style.height=window.innerHeight+"px";	
	};

	var init=function(){
		cvs=document.getElementById("theCanvas");

		// Set width and height of our game raster
		// (in pixels). These will never change
		cvs.width=900;
		cvs.height=500;

		resizeCanvas();

		ctx=cvs.getContext("2d");
		ctx.font="20px Bold Arial";
		ctx.textAlign="center";
		ctx.textBaseline="middle";


		var mainLoop=setInterval(function(){
			// Loop code goes here
			//
			frame++;

			ctx.fillStyle="#0f0";
			ctx.fillRect(0,0,cvs.width,cvs.height);
			ctx.fillStyle="#000";
			ctx.fillText("Frame "+frame,cvs.width/2,cvs.height/2);

		},20);
	};

	// Public properties
	return{
		resizeCanvas:resizeCanvas,
		init:init
	};	

})();

So that’s the most basic HTML and JS code. If you run it on your browser (computer or phone) it should show a green background with the words Frame XXX in the middle with XXX increasing rather fast. The good news is the HTML code is done!. That’s it! same with the CSS! after this all we’ll be working with is Javascript and the only element we’ll be working on is the canvas.

In part 2 we’ll “plug in” our gamepads and get them live and responding to keypresses/touch events. Stay tuned!