Interviews: Andrew "bunnie" Huang Answers Your Questions

Interviews: Andrew "bunnie" Huang Answers Your Questions 32

Posted by samzenpus on Thursday August 21, 2014 @11:32AM from the here-they-are dept.

A while ago you had a chance to ask Andrew "bunnie" Huang about hardware, hacking and his open source hardware laptop Novena. Below you'll find his answers to those questions.

Why "bunnie"?
by Anonymous Coward Seriously, that's the first question I have whenever I see his name. I just can't get past it. What's the story behind that nickname? Please tell me so I can focus on his good works instead.

bunnie: “bunnie” is a shortened version of my full nickname, which is “vorpal bunnie”. In Monty Python and the Holy Grail, the vorpal bunnie is an innocuous-looking rabbit that would suddenly spring forth, decapitating King Arthur's knights with its sharp pointy teeth. It was also a mob in the Rogue clone “Moria”, and it had the interesting property of exponentially reproducing if you didn't kill it immediately. Whenever I ran into it, I would kite it around the dungeon until the level was overrun by vorpal bunnies.

It was around this time that I had to pick a handle for logging in to BBSes and I don't mean punbb, I mean the old 1200 baud dial-up jobs from the 80's. My friend helping me set up an account was probably just trolling when he recommended “vorpalbunnie” as a username, but lacking any other inspiration, I ran with it.

When I enrolled at MIT, it was 1992, and the Internet was still just a curiosity. I had to pick a logon for their computer system, Athena. I hated the generic-sounding default of “ahuang”, so I went with my BBS nickname. I had to shorten my username to bunnie because vorpalbunnie was too long – back then, usernames and passwords were often required to be less than eight to ten characters, rather than more.

Still, few people called me by bunnie, because well, the Internet wasn't very popular and aside from a messaging client called Zephyr, nobody saw my username. This all changed in 1998, when I was a TA for a required Computer Architecture class, and the professor in charge couldn't remember my actual name and he introduced me to about 300 students as “bunnie”. The name was unforgettable – nobody would remember Andrew once they had heard bunnie. I realized at this point I wasn't going to win against the nickname, and so I accepted my fate and henceforth adopted “bunnie” as my new identity.

Chinese industry
by Esteanil

What is the most important thing you wish you had learned earlier about manufacturing in China?

bunnie: Chinese. And how to use a squatting toilet.

Re:Chinese industry
by tchdab1

And speaking of learning, where have you learned things that you could not easily or ever have figured out on your own? Which environment or learning experience do you look back on with gratitude?

bunnie: This is an interesting question. There's a lot of things I couldn't have learned on my own. I don't think I could have taught myself physics; there is in fact a place for classroom learning. I don't want to overemphasize the value of textbooks, but a certain amount of textbook study is necessary, particularly in the hardware world, and having teachers who can explain difficult concepts to you through simple analogies and/or who take the time to understand what you're missing and give you the clues you need is important. Hardware is based upon physical principles and having a solid foundational knowledge of physics will help tie everything together.

That being said, book learning alone won't get you all the way there. There's a lot of practical hands-on knowledge that is not taught in classrooms but I have acquired through many mentors. Generally, I like to surround myself with people who are smarter than I am; you learn a lot just by observing the methods and manners of these people. Subjects like how to debug circuits, running manufacturing operations, how to manage people, how to negotiate, and how to hack are not easily figured out on your own. I like to watch people do these things, and learn from their techniques.

I think the key thing about the environment is that the transfer of knowledge really only happens in a situation where things are “being done”. In other words, you don't learn much from conferences, presentations, or lectures on these topics. I always find it a bit of an irony when I'm asked to lecture on things like manufacturing or hacking. When you can sit shoulder to shoulder with another engineer and watch them problem solve, that’s when you can really see their thought process. You can see all the failures they go through, and the twisting path they take to finally arrive at the answer. In a lecture or presentation, there tends to be a focus only on presenting the problem and the solution, but none of the intermediate failures. Those intermediate failures contain very valuable lessons, which is why hands-on learning is so important.

I'd say my “real” learning began in my PhD program, where you only had to take the courses you really needed to understand your research, and you could skip all the fluff. Also, in that environment, you got to sit with a bunch of peers who are all struggling with difficult questions, and you share notes about how you fail every week, and trade suggestions on how to improve; and you have the wisdom of a thesis advisor who can help guide you when you're really lost. The nice thing about starting my “real life” in the PhD program is that it was an environment where I could take huge risks, fail, and the only consequence is I'm set back on my graduation date. Once you step into a startup, you're burning capital and you're working to a deadline; you become much more risk-averse. That's probably the biggest difference I see between people who went straight to industry vs. graduate school: folks in industry are great at execution, but tend to stay with what they know. Folks who went to graduate school can be unreliable and late for deadlines, but they also tend to take more intellectual risk.

How do you become a professional hardware hacker
by werepants

What advice would you give to a person who wanted to make a living in the "Maker" tradition - being able to spend your days designing, engineering, and building on technically interesting and creative maker projects? I'm most interested in the career aspect, assuming that you've already obtained a preliminary education: would you look for a job with a similarly minded engineering firm, launch a kickstarter, start a hackerspace, hack together some things and try to sell them through a webstore, work as a freelance engineer, or something else entirely?

bunnie: Hardware hacking is a lifestyle. You sort of end up living in it – unlike software people, where all your code is stored in the cloud and all you need to perform is a laptop, hardware people accumulate boxes and boxes of parts, and need a lab full of equipment to get their work done. Novena is an attempt at consolidating some of that into a portable form factor but I'm never going to be able to give up a work bench equipped with a good Tektronix scope and a soldering iron.

Previously, I thought there were two career options in life: either go corporate, or go startup. There's a third path that's not often talked about – the “lifestyle business.” VCs and corporates ridicule this option, because its goal is to make enough money for you to sustain a lifestyle, but doesn't have a “future” – it won't scale to a billion dollars, it won't be acquired by a big company, and it probably won't ever IPO. This does not mean, however, that you have to live on instant ramen and bother your parents every year for money. Lifestyle businesses can grow to million-plus per year turnover businesses, and the margins can be quite good; so when you run the math, your net cash flow can be better than a corporate day job.

The key to starting a lifestyle business is to find a problem that someone has, and to solve it. This sounds really simple, but in fact in today's world of mass-manufactured solutions, what you find is you have cheap solutions that get you 90% of the way there, but few people have 100% solutions to their most pressing needs – and that last 10% is where all the difficulties lie.

A first-order test for whether you've found a genuine solution to a real problem is how much someone would pay for your solution. If someone has a real problem and they really need it solved immediately, and you can convince them you're going to actually solve it, the customer will typically pay a premium for that solution. I always wondered, for example, why there are equipment companies that can get away with selling specialty products that constitute not much more than a panel meter with an Arduino in a badly injection molded plastic case. Certainly, a multimeter can be a potent tool in the hands of someone with a degree in electrical engineering, but a measurement solution that you can hand to any untrained field worker and get correct, consistent results that you can rely upon is hard to come by. Compared to the cost of paying someone to drive out to the field and repeating measurements day after day, it's a steal to get a foolproof piece of kit for a few hundred bucks. In other words, it's not the value of the parts the customer is paying for – it's the value of getting it right the first time and every time after.

The biggest down side of a lifestyle business is that it's not consistent – typically you have one or two major clients, or your clients all concentrate in the same business area. This means your revenue can be highly unpredictable and your business will rise and sink with the macro-economic tides. Practically speaking, this means you need to stay debt-free – once you take on a mortgage or saddle yourself with a fancy car you bought with borrowed money, it's hard to accept the risk of running a lifestyle business. This is easier said than done; there is a lot of peer pressure to keep up with the Joneses, at least in America. One of the subtle benefits of living in Asia is the locals are cheap – the old money tends to understate their wealth and many millionaires ride the bus and eat regularly at the local food court to save a buck or two. It also doesn't hurt to to have a safety net of friends and family you can rely upon – knowing you can always call up your folks or having a sugar mama who can reliably cover rent during thin periods helps mitigate the risks of running a lifestyle business. Finally, it helps to take these risks before you have children, or simply decide to not have children at all – your priorities change once you have dependents who rely entirely upon you for their livelihood.

100% Open Hardware
by Anonymous Coward

At what point will we be seeing a 100% complete open hardware platforms, replaceable~ for modern OTS offerings? By that, I mean from silicon manufacture to FOSS binary. 100% open design, manufacture, and source code. I'd like to think this endeavour isn't more than a thought experiment.

Small Scale Chip Manufacturing?
by Cpt_Kirks

Where do you see small scale chip manufacturing, up to and including custom multi core CPU's, going in the near future?

bunnie: Combining these two questions into one answer.

There is a dirty secret about open software vs. open hardware. In open software, bits are virtually free to copy, store, and distribute. In contrast, for open hardware, atoms are all owned by someone or some organization; or if they aren't specifically owned (e.g., air, water, minerals in the ground), the access to said atoms are regulated by governments or warlords (in regions with no functioning government).

And thus, as you peel the onion of openness, you find yourself asking questions such as, “who owns this pick and place machine?” “who makes the lithographic stepper?” “how was this copper smelted?” “where did this tantalum come from?”. Why should F/OSS simply stop at silicon manufacture, after all? Or for that matter, why do people care so much about F/OSS chip manufacture, yet have no problem with the closed-source processes and machines being used to manufacture PCBs? At the end of the day, there's a fancy multi-million dollar tool that's being used to write the mask patterns for your IC or your PCB, and do you trust that to not have a back door? That machine also probably runs some flavor of Unix, or failing that, is probably loaded with GPL software that's not being used according to the license.

As a fun mental exercise, I looked into what it would take to make an MCU with the power of an Atmel AVR to replace the closed-source IC that's inside an Arduino. Currently, it seems you can design the Verilog hardware description, synthesize it, and lay out the logic using a fully open source tool chain (although you'll probably spend more time managing and fixing the design tools than you'll spend designing with them). However, you would still have to sign an NDA to get access to the process geometry files, and the EEPROM memory block is provided as a drop-in hard macro. As in, you don't get the source for the EEPROM, you simply leave an EEPROM-sized hole in your maskwork and the fab will slot in the respective polygons for you (there's a long list of plausibly legitimate reasons for why this is the case that I won't bore you with). So if you really wanted to insist that all the design source was open, you couldn't build something that could retain its program memory without power, which is kind of essential for a self-contained MCU like an Arduino.

And then, if you think through this exercise a bit further, even if I shared the high-level design source (Verilog, schematics, layouts) with the world, if users can't access the design rules for the fab process, they still can't practically modify the design; and even if they were okay to sign an NDA to get the process design rules, they'd still have to fork out tens of thousands of dollars to get samples made.

So at the end of the day, the idea of having a BSD-style “make universe” in hardware is tantamount to writing a manual for rebooting civilization from the ground up to the point of manufacturing silicon.

The open hardware community has struggled with this reality, and the best proposal I've heard to date is the concept of “layers of openness”. Hardware is built upon very strong and modular abstraction layers; it's not like software where you could, technically, reach around a virtual machine and tweak real machine parameters if you had to. Silicon is silicon, PCBs are PCBs, and although there are research projects to merge the two in practice there is a very bright and clear line dividing those two abstractions thanks to things like chemistry, physical laws and supply chain logistics.

While we never got as far as making iconographic labels for each layer of openness, there is an idea that well, it's sure better that the PCB design source is being shared than kept proprietary, even if the ICs are closed and the PCB manufacturing process is closed, and the design tool for the PCB is closed. So generally, the hardware community's consensus is that it's legit to say hardware is “open” if you're more open than the status quo; but in reality, if you look at the total amount of human design knowledge, capital investment, and manual labor that goes into making a piece of hardware, we're going to have to rely upon a complex web of suppliers and sub-contractors driven more strongly by issues such as cost management rather than openness for quite some time.

The good news is that as Moore's Law grinds to a halt, fab processes are becoming fully depreciated, dropping the up-front cost of doing a new IC design by orders of magnitude. This means an “open source” (at least at the level of HDL source code) CPU is probably just a couple years away. In fact, there is a team at the University of Cambridge that I'm advising that is making an honest run at building a usable open source CPU. I think their team has a great chance to meet their goal and I think it's definitely a step in a right direction. Their first run at it won't have the performance of top-notch mobile CPUs, but their current direction may include some very unique and meaningful architectural features that will set it apart from closed source alternatives when it comes to security and code reliability.

How was it growing up?
by haneefmubarak

How avid of a hacker were you when you were in high school and how supportive do you feel your friends and family were of your hobby?

bunnie: I started hacking – or rather tinkering – with electronics around the age of 8. By the time I was in high school I was a hopeless nerd (now that I think about it, I was a nerd before being a nerd was cool). High school was actually not too bad, because I went to a magnet school with other nerds, and my teachers and friends were very supportive.

The toughest times were in elementary and middle school. Kids that age have no concept of political correctness, and being both a meek nerd and the only Chinese boy in a midwest school set me up for a lot of bullying and ridicule. I had a couple of other outcast/nerd friends who I'd hang out with occasionally, but mostly, I just kept to myself. I'd just sit in the back of the cafeteria playing with my programmable calculator and reading datasheets (believe it or not, I would carry around as easy reading electronics catalogs and three-inch thick databooks published by Intel and other manufacturers). Most kids didn't even care to understand what I was looking at; they were more concerned if I could see through my squinty eyes and having fun flicking my large Asian earlobes so they would swell and become even bigger and more goofy looking. But being an outsider has its advantages; I was less constrained by the social pressure other kids felt, and overall I was having a lot of fun building hardware.

My parents were always extremely supportive and patient, and they often turned a blind eye toward things I would take apart but not quite get back together into its original fit and form. I think they were happy enough that their son would rather spend summer days in the basement hacking on the Apple II and reading volumes of “Getting started with Science” rather than chasing girls and smoking cigarettes stolen from the local convenience store.

Restrictions in the future?
by plover

Do you see manufacturers of the future attempting to put restrictions on hardware hacking, either more technical or legal? Will manufacturers order CPUs without I2C pins, or toy drones with UEFI secure boot operating systems? Have other countries put restrictions on hardware hacking that have affected you?

bunnie: I don't see pure hardware manufacturers putting restrictions on this. Manufacturers actually want lots of back doors and flexibility to probe their hardware, because they have to be able to debug or reprocess parts that fail to pass quality inspection. Scrap rate directly impacts the bottom line, so there's a strong financial incentive to make hardware that's easy to debug, and transitively, easy to hack.

The people who tend to put restrictions on hacking hardware are people who own content or run services on the hardware. Hardware in that case is seen as a barrier or a method to ensure lock-in to a particular platform, or to require payment for services. Occasionally, content providers are the same as the manufacturer, in which case you tend to see a lot more restrictions put in place on the hardware. As long as the hardware world isn't taken over by the Apples, Amazons, Microsofts and Googles of the world....oh wait, nevermind. We'll just have to keep on defending our rights as consumers by jailbreaking and hacking for some time to come.

But if someone is purely in the hardware business, they actually have no incentive to lock you out – they'll make all the money they ever will selling you the hardware in the first place, and if you bought it just to tinker with, that's just one more sale for them.

How do you go about discovering hacks?
by Joe Gillian

I haven't read your book (I will when I get off work) but I'm curious as to how exactly people discover these hacks. I mean, there's some really weird ones out there that make me question how people even thought to do them, such as hacking a PSP battery into service mode in order to load custom firmware or manually opening a PS2's disc tray to bypass the copy protection that only activated when the button to open or close the DVD drive was pressed. I know with the Xbox, there was a software hack (I don't know if it's the same one you found) with save files from certain games, but only specific versions of those games. So my question is, how do you go about looking for exploits?

bunnie: Surprisingly, many exploits are based on well-known holes or common mistakes made during implementation; it's almost as cliché as kicking in the door with the “unpickable” electronic lock. In hardware, there's almost always JTAG diagnostic ports or a serial console available, and most manufacturers don't bother to plug them. In software, there's a lot of complexity and things tend to leak at the seams, so poorly implemented APIs that lack bounds checks is a very common avenue for exploits. I actually don't think I've “invented” any exploits in my entire career; everything I've done you can find examples of in prior literature, or they are so obvious that one cannot really claim credit for it (a bit like climbing in through the window when the front door is locked – it's just common sense).

But generally, the first things you tend to look for are backdoors put in place for diagnostic or engineering purposes. I'd say 80% of the hardware I've hacked have welcome mats like this set out for me. Then, you look for opportunities to gain access to the code image within, either through dumping ROMs or injecting a small program that can dump RAM. If you can get the binary code, you can reverse engineer it and look for exploitable APIs and function calls. Lacking such code, you can attempt to characterize the system through fuzzing and scanning the available APIs; this involves, for example, injecting garbage into file structures, sending malformed packets or glitching wires and clocks to get the hardware into a vulnerable state. And then there is the whole host of somewhat more difficult vectors such as microscopy and side-channel analyses through power signatures and parasitic emissions, but you usually fall back to those only once all else has failed. There are also occasions where you are confronted with a CPU with an undocumented instruction set or a lack of documentation on its registers or interrupt handlers. In this case, it really helps to have an automated fuzzing framework because often times this boils down to doing a brute-force search of memory space for certain signatures and signs of the registers or behavior you're looking for.

And, one should never pass up trying obvious things, like googling for the root password included in the SoC's maker's devkit and trying that on the diagnostic console. This has worked surprisingly often...

Hacking the Xbox
by nmb3000

One of my first forays into the realm of hardware hacking was following along as you recorded your exploration of the original Xbox console. I was fascinated by the hardware, but enjoyed your analysis and methods even more. It was you that got me interested in hardware and hacking. (Aside: Thank you very much for releasing your book as a freely-available download and for the open-letter about Aaron and MIT)

What was the most memorable experience for you of your Xbox expose? Was there a particular part of the hardware that you found especially well-designed (or laughably poor)? A method that yielded unexpected success (or failure)? What kind of fallout from Microsoft did you face? I remember you posting the voicemail of the Microsoft employee asking you to remove the images of the Xbox ROM -- something I got a good laugh out of. And as a follow-up: do you have a feeling for how "secure" hardware has changed in the decade since the original Xbox launch?

Thanks for taking the time to answer our questions, and also for all the work you've done pushing for a world with both open software and open hardware.

bunnie: Thanks for reading the book! Actually, the book pretty comprehensively documents my experience hacking the Xbox, even a decade on there's no new secrets for me to give out. There's always that massive endorphin rush when your experiment works out – I remember the night I found the secret key I could barely sleep; I was trembling with excitement.

But it's also important to remember there's a lot of luck involved and at the end of the day I really prefer to do hacking for fun, and not as a business model. It's much easier to carve out a living, in my opinion, building things than breaking into things; sometimes you encounter things that are really difficult to break into and you end up expending much more resources and time than the contract is worth. So, I find it hard making money as a hacker: I either charge too little for the service, or I lose money completing a difficult job that I underestimated.

I think Microsoft has done a very professional and comprehensive job securing their latest generation of consoles. They hired some of the best and the brightest hackers to design it and they actually listened to them. You'd also be amazed at how incredibly hard it is to get management to sign off on silicon modifications to improve security, but these days more and more manufacturers are starting to “do it right”.

This discussion has been archived. No new comments can be posted.