We all know that modern computers operate on binary numbers and are extremely efficient at doing so.
But this was not always the case.
But we are getting a little ahead of ourselves.
In this article, we will take a look at how this all came to be.
The ability to perform computations on numbers expands their utility beyond just counting things.
This was precisely the question Gottfried Leibniz spent his life trying to answer back in the 1600s.
In the field of mathematics, he is most famous for his independent invention of calculus alongside Isaac Newton.
Fast forward to the 20th century, and this would actually become the fundamental tenet for binary computers.
Mathematically, this is akin to using negative exponent values.
As a quick primer, our “natural” way of using numbers in everyday interactions uses base 10.
With this methodology, we can encode any arbitrary value using our set of predetermined symbols.
In the binary system, there exists only two symbols: 0 and 1.
With just these three operations, Boolean operators allow for a foundation to use binary to process information.
And of course, with 32 bits, you’re able to represent up to 4,294,967,296 unique things.
What are these “things”?
This limitation turns into a hardware problem, since numbers are fundamentally limitless and infinite.
Herein lies the fundamental idea behind hardware number representations.
The “Problem” with Numbers: 1970-1985
Numbers are infinite in nature.
From an information theory perspective, the closely related question of, “Howmanynumbers can be represented?”
is tied to the number of bits available.
Going back to our example above: suppose you choose to represent numbers using 8 bits.
That means you’re able to represent up to 2^8 unique numbers, or 256 numbers.
Which two-hundred and fifty-six number you choose to represent is a different question.
Furthermore, what do you do with the end points?
In the last example, do you choose to represent 0 or 1?
You don’t have enough bits to represent both!
With 8-bits, it’s possible for you to represent up to 256 unique values from0000 0000to 1111 1111.
Another challenge is how do you handle “weird” situations, such as division by zero?
In the hardware, do you want that to be represented as “infinity”?
Or maybe reserve a bit representation for “Not-a-Number (NaN)”?
Which unique bit sequence do you set aside for these “denormals”?
What do you do?
In the 1970s and early 1980s, this led to the wild west of number formats.
The IEEE-754 floating point standard was established in 1985 to address this.
Specifically, code portability helped usher in the adoption of this standard.
(We still haven’t addressedwhatthat outcome would be and what approximations IEEE-754 would make standard).
The details of the refreshes are way too technical, but you might check out theWikipedia articlefor specifics.
Accolades aside, whatisFloating Point?
Let’s walk through this.
Additionally, there are certain numbers within this range that are not representable.
The value of 21845 is not even close to being represented!
To pull this off, additional hardware is required to reinterpret the 32 bits, as shown below.
Recall the discussion about precision versus range?
Let’s put this all together!
How would we represent 5 in IEEE-754?
The sign bit is 0, since this is a positive number.
With that, 129 - 127 = 2.
The 1 is implied, leaving us with010 0000 0000 00000000 0000.
Our final 32 bit representation is0100 0000 1010 00000000 0000 0000 0000.
Ok, well, that wasn’t as straightforward as someone would assume.
To save you some time, here is ahandy calculatorto play around with floating point numbers.
This allows subnormal numbers to represent very small values with limited precision.
This was an implementation issue in the Pentium processor.
Besides that, manyjokesand memes among computer science practitioners arose during this time.
However, in the last decade, the standardization of number formats hit a road bump.
And why would they: it had solved one of the biggest problems with early computing: portability.
This allowed for easier packaging of applications and maintained a semblance of consistency in machines worldwide.
And one particular hardware optimization is changing the precision and range of numbers for DNNs.
A logical optimization then is to reduce the number of exponent bits and mantissa bits.
Luckily, there already exists such an optimization in the IEEE-754 standard, called Half Float.
A Half Float is exactly that: 16 bits instead of 32 bits.
From an utility perspective though, fewer bits did reduce the DNN accuracy.
What else could be done?
Coming out of Google Brain, they aptly named this new format Brain Float, or BFloat for short.
Other number formats soon followed from other companies, including IBM, Nvidia, and AMD.
Why stop at just changing the meaning of numbers in hardware?
What if you’re able to make a few hardware changes to make things run even faster?
This is the path Microsoft took with Block Float.
Said differently, the values don’t span a large range.
The figure above illustrates how block float works.
Its resurgence today mostly has to do with the domain-specific hardware optimizations, targeting AI.
Another cool number format that recently emerged is called AdaptivFloat.
Values within a layer typically do not span a wide range themselves, but across layers they could differ.
How does AdaptivFloat accomplish that?
By adaptively changing the exponent bias in the floating point standard.
Recall that in the IEEE-754 format, there is an implicit bias of 127 applied to exponent values.
What happens if you change the implicit 127 to something like 125?
AdaptivFloat is a clever little hardware trick that is definitely software-inspired.
), resulting in various versions such as CFloat16 and CFloat8, as employed by Tesla.
The final number format we present are posits.
Compared to the traditional FP32 standard, posits have an extra field for the regime.
They also play a crucial role in the accuracy and performance of arithmetic operations performed in the posit format.
Posits show immense potential, especially in scientific computing applications.
The impact on code portability will also be interesting to follow.
How will this impact future processor and accelerator design?
It will be exciting to see where things take us in the next 10-15 years in this arena.
What’s in a number?
Well, a lot more than what appears at face value, for sure.
