Oh, don’t fear, I’ve screwed up plenty of things too
Oh, don’t fear, I’ve screwed up plenty of things too
I’ve gotten again into the swing of writing once more for the same old
and Hacker News has observed. Yesterday’s publish bought on there and
got here out, far and wide.
But, hey, even in turds you’ll be able to typically discover a peanut, and one of them
mainly mentioned “she never talks about things she’s screwed up”. I
thought I had, however with over 1200 posts, I can see the way it may be arduous
to seek out some of it.
So, simply to make it simple, I’ll attempt to rattle off some tales
about things I broke sooner or later in time. Prepare for optimum
Thirty years in the past this month, I began a public BBS. That is, it had
its personal devoted cellphone line, and members of the general public might name it.
It was now not only a testing factor I ran on my household’s single
(voice) cellphone line for buddies to play with.
While up within the attic hooking things up, I wanted to check one thing and
so clipped onto the primary line to ring the second. I used to be nonetheless holding
the second line on the time… and bought to seek out out what 90 VAC at 20 Hz
looks like. BZZT. Not my brightest second.
Later that afternoon, as soon as it was all hooked up, my first new person
known as in… virtually. They bought related, and when it requested them “person
id or NEW”, they typed in NEW, and it hung up on them. Why? Because I
had put the factor in “no new users mode” when it was on the voice line,
“just in case someone else found it”. My first likelihood at having an actual
person, and I instructed them to take a stroll.
It was three extra days earlier than anybody else known as. Oops.
I believe I plugged in my modem (a Commodore 1670) upside-down as soon as. I
don’t keep in mind how. It in all probability concerned being careless and never wanting
at what I used to be doing. It blew the fuse on the machine and introduced the
BBS down for some time till I might discover one other one.
Back after I was serving to wrangle cat footage, I had a starter venture to
redo the file cache in HHVM. (I can use some specifics right here because it’s
open supply. Really. Go look.)
I bought all the pieces working and flipped the swap, and promptly broke the
subsequent push. Why? I had conservatively
the file mode used with open() to 0600 – person learn/write, and nothing
for anybody else.
Trouble is, they constructed the cache with one person id, and ran the precise
server with one other. The person id was preserved when the cache was
distributed to all of the net servers (suppose tar’s -p swap). So the
first time they tried to learn the new-form file, they hit -EACCES and
The RelEng people chided me and labored round it by including a chmod to
their launch script. Then I
the precise mode upstream to make it 0664 from the get-go, and
then hopefully they eliminated the now-superfluous chmod.
My first time on name for mentioned cat footage, I bought a request from
somebody to attempt to discover a sample of their core dumps. The frontend
machines in query ran some helper job and it was falling over. This
place used to truly symbolize core dumps and left a .txt file in
/var/crash subsequent to the precise core.
I figured, okay, cool, let’s simply run grep on all of these .txt information on
the cat image frontend machines on the market to see if we are able to discover what
they had been on the lookout for.
parallel_ssh_thingy -S catpix.frontend.region1 "grep foo
I let that run for a bit and turned my consideration to different things. I
figured I might come again to it in 20 minutes or so when it had hit all of
A couple of minutes later, I observed that cat image uploads had gotten VERY
VERY gradual in a single area. Like, actually unhealthy. I opened a SEV – my first
one – to trace no matter it could turn into.
I bought on a gradual field. It had this huge grep operating. It was wanting
for a string in each single file in /var/crash, together with the
a number of tens-of-gigabytes-large core dump information. This was
beating the dwelling crap out of the disks, and was slowing the machines
down sufficient to make the monitoring sad.
I killed the greps and things went again to regular.
Thus, the primary SEV I ever opened within the instrument was additionally the primary one I
brought on. Derp.
Every firm appears to write down one of these “run things in ssh all over the place
without delay” tools sooner or later. They should all be renamed “immediate SEV
One time I used to be working in a basement of a home on some telco wiring.
This home was getting a “proper” set up with precise punchdown blocks
and all of that great things. Something wasn’t fairly proper with the
block, so I used to be crammed right into a nook within the utility room subsequent to the
furnace twiddling with it.
That’s after I observed that each time I touched the punchdown block with
the (all steel) instrument in my proper hand, my left ear tingled. It was
repeatable and distinct: contact equals buzz, take away equals not.
It did not take too lengthy to place it collectively: the road was sizzling, so the
block was sizzling, too. It was simply the same old on-hook line present, however it
was nonetheless notable. It clearly went throughout the instrument and proper up my
hand no downside.
As for my ear? I used to be leaning towards a steel publish which held the home
up… and which had been pushed down by means of the slab into the (literal
and electrical) floor. So, that is the place the present went.
I designed my authentic
“mail delay” system
notably badly. It used a database schema which amounted to a row
id, after which strings for every of the supply IP tackle, HELO string,
FROM string, and TO string. It took up a silly quantity of house on the
disk and was tremendous gradual, too.
It bought higher. But that preliminary model was crap.
Many years in the past, I used to be coming off the freeway at an uncommon place as a result of
my ordinary exit was closed. I could not actually see too nicely to my left,
however took the best on pink anyway as a result of it appeared “clear enough”. It
wasn’t. I needed to dive for the shoulder as a automobile shot by and honked
(fairly rightly) at me.
That one was notably silly, and I am going to always remember it.
Back within the days when the servers and swap at my sysadmin job sat on
my precise desk, I used to be fixing some wiring and managed to hit the large
ON/OFF toggle on an influence strip which killed the swap, and thus
interrupted any telnet classes in use, together with the one utilized by the
boss. I turned it proper again on, however it took a number of seconds to start out
forwarding frames once more. That was sufficient to hose things.
CLOMP clomp clomp clomp clomp. I might hear him coming down the corridor.
Before I might even crawl again out from underneath the desk, the boss was at
my desk asking “WTF?!”.
I now put a mollyguard over these things any time there’s any likelihood of
them being uncovered and having unscheduled activations.
I used to be residence alone as a child, watching some film on TV. I noticed some man
seize a beer can and do this factor the place you jam a pen within the aspect to
make a gap, after which crack open the highest. The concept is which you could
suck the entire thing down without delay as a result of the opposite opening lets air
stress equalize, and so it would not cease to “breathe”.
I went into the kitchen and grabbed a can of soda and a pen. Then I
jammed it into the aspect and … FWOOOOOOOOOOSH… it shot up and out and
all around the kitchen, INCLUDING the ceiling (which I had no likelihood of
reaching to wash up to cowl my idiocy).
a binary tree implementation in plain previous C that turned out to
be a linked listing in observe as a result of all the pieces ultimately ended up
sorted, and it could simply chase down the best aspect pointer.
It simply stored getting slower and slower till I lastly went and located
out why. Oh. Yeah. That would do it.
(I am going to notice that after I wrote about this six years in the past, THE ONE confirmed up
and asserted they might do no such factor. I assume THE ONE would not.
But the remaining of us would.)
Back within the days once we constructed small Linux containers on leftover/junk x86
to behave as dialup routers for distant websites, I constructed one for the
“alternative” highschool – for youths who bought pregnant or had hassle
with the legislation, or no matter. That far again in time, I did not all the time belief
LILO, and tended to make a boot floppy that will carry the field up simply
This one was no exception, though I ended up getting LILO to work, and
so a subsequent kernel replace in the course of the construct of that field did not exit
to the floppy. One necessary factor I had switched on was PPP assist.
The arduous drive had it, however that floppy did not. I did not actually
recognize that on the time.
This did not matter for a very long time as a result of the floppy merely sat IN the
floppy drive loosely however was not really pushed all the best way in. That
is, it wasn’t totally engaged and so the drive could not see it. It was
simply chilling as an “in case of emergency” sort factor.
One day after an influence outage, I came upon their machine did not get
dialed again in. I went on the market anticipating the worst, and located that
somebody had pushed within the floppy. They will need to have observed it “half in,
half-out”, and pushed it in the remaining of the best way in. Then when the ability
flipped and flopped, it rebooted off that disk, and located itself unable
to face up a PPP hyperlink connecting to the skin.
I needed to drive all the best way on the market as a result of I left a sexy
nuisance there simply begging to be “fixed”, and moreover, the on-floppy
kernel lacked the ONE factor the field wanted to be remotely accessible.
That entire college was offline for many of that morning.
Yeah, that is sufficient fodder for right now.