[Music]
hello and welcome back to the open
hardware mini conf
now according to the original schedule
steph piper was going to be doing a talk
now but the amazingly talented stuff is
unwell at the moment so i'm filling in
and
what i'm going to be talking about is
when things go wrong so
this whole presentation
could fail terribly and i will still
have succeeded i hope that's my theory
if i set out to do a talk about failure
and it fails then that's a win
so
there were a few difficulties that we
faced
oh and also just a reminder if you have
questions and things i'm not necessarily
going to see them live but
drop them in the chat
and
mark said
did i steal his slide of
if you didn't design a fuse in your
circuit the circuit will find one for
you
yes that is actually failure number two
we're going to get to that mark we have
a good example of that
so
uh
this
project and this collection of projects
i should say because there for the
hardware project this year we really had
three
loosely associated boards there is the
swag badge
the rockling which is the sailor that
goes on top of it with the fpga and the
party button which steph designed
and we've had issues with all of them so
i'm just going to talk about a couple of
the things that we faced
and
ways we worked around
those issues
so i'm going to try to show you some
live stuff
i have zero slides it's just going to be
looking at hardware and messing around
with it and if we're really lucky we'll
let some smoke out
so
uh where to begin okay so
the um
this is the rockling which is the little
fish-shaped fpga board with the
uh the theremin front end on it which
was the
outline of this was designed by andrew
and the pcb layout was done by bob who
you can see over my shoulder on
this side
and so
there are some very interesting aspects
to the design here
which caught me out when it came to
assembly
so the first thing is that because this
is being designed
primarily well not primarily but one of
the major aspects to it is the visual
the way the board was designed we wanted
this to be the front that is presented
to you when it's in use
unlike most pcbs we didn't want all of
the parts to be on the visible side we
wanted them to be on the back
so andrew designed the layout of the pcb
and then bob did the layout and putting
all the parts on the back and he's going
to talk about that
right at the end of the mini conf so
that'll be really interesting to see as
well
but one of the implications of that
is that when the production files are
generated out of keycad
it has all of the parts on the back of
the pcb and not on the front
now that's not really a big deal as long
as you are paying attention which
apparently i was not
and
it led to a bit of a problem
so what happened was that the production
files were generated
and i uploaded the gerbers
to jlc pcb who did the fabrication of
them so it's a four layer pcb
and on the order form for the pcbs there
is a little tick box where you can
select whether you want a stencil
so the solder paste stencil is used to
used as a mask to control where solder
paste is applied to the board
i ticked that box didn't think any more
of it
and selected the solder mask
for the top of the pcb
and when it got back when it arrived i
opened the box and this is the stencil
they shipped me
you may notice there's something missing
a little thing called apertures apart
from a couple here which are apertures
that are on the bottom of the pcb
all of the important stuff all of the
apertures that we need
for applying the solder paste
but just not there
it is entirely my own fault
oh and
we're back from that little video
failure it is entirely my own fault
for
not being aware when i was ordering it
now the thing is that these production
boards we were starting to run a little
bit short on time so there wasn't
time to simply order a new stencil
luckily bob remembered that we still had
the stencil from the first round of
prototypes
so
right at the start of the process we had
ordered a small number of prototype pcbs
and on that order the stencil was on the
correct side of the board
and this is the stencil right here so
this is the stencil for the prototype
now the difficulty is that the stencil
for the prototype was not quite the same
as the stencil that we needed
so it's not just a simple matter of
putting it down putting on the paste
because some of the parts had changed
and a couple of the parts had moved
so
we then had to try to figure out how to
get the stencil and how to get the
solder paste onto all of the boards in
an efficient way
and
one thing i had heard of a story i'd
heard many years ago
was about people modifying solar
stencils by
stencils by soldering closed apertures
that they don't want
and so i thought why not give it a try
however i very quickly discovered that
it is pretty much impossible to solder
stainless steel
and when i tried i tried different
fluxes and things i just couldn't make
any solder stick i tried abrading the
surface
probably with the right flux and the
right solder i could have made it work
and i could have soldered closed the
apertures that i don't want anymore
but i failed
so there was then a suggestion of how
about just covering up apertures with
tape and that's what i'm going to show
you now so you can see on the back of
this stencil
there are some of the apertures closed
up with kapton tape which is very very
thin
i was quite skeptical when i first heard
of the idea
and i thought well let's just give it a
try we'll see what happens
what i was worried about was the
thickness of the state the tape holding
the stencil off the pcb and allowing
leakage underneath
so
if i
spin over here and go to the overhead
camera
you can see here
oops the the stencil
and on the back
you can see the kapton tape
so what i did was use a scalpel and some
tweezers and cut off tiny little pieces
of tape and then just covered up the
holes that i did not want paste to come
through
so that way it
it wasn't putting paste down on the
board where i don't want it
and what you can see here is
the um the vacuum bed this system was
designed by ceon the unexpected maker so
it's a really cool system the idea is
that you connect a vacuum cleaner and
this acts as a vacuum bed and sucks the
stencil down
onto the target board
so and there is a cut out there
this particular cutout was done by
exporting a dxf
out of the original design that andrew
did
and then bringing it into fusion 360
applying the dxf as a shape onto this
and then
recessing it enough for the pcb to fit
in there
so what i was thinking of doing if i
get this in approximately the right
place now this is not at all how this
would normally be used
this is just sitting on my bench because
i happen to have an overhead camera here
and please excuse the back of my head
but what i'm going to do
is
just put some solder paste on here
and
show you the end result
so
this is not actually going to be
populated
so i'm not really all that concerned
this is just a test of the whole process
of
applying solder paste with some of the
apertures masked out
so grab my spatula
scrape it across the stencil
i'll just put that aside for now and try
not to lose any of the expensive solder
paste
the pcb is stuck on the bottom there we
go it's fallen off
so there we have the pasted pcb and
because i was just doing this without
any proper alignment
i suspect that these apertures are going
to be all over the place
but at least you can see there under the
microscope
we'll get the focus correct
that there is paste on the pcb
and if i
move over
where is an area
well you can see there are some examples
these particular
pads
just here don't have any solder paste on
them because i masked that out on the
stencil
and you can see the alignment there is
quite bad but also there are some parts
down here that have no solder paste
and down here
there were also some parts that
were on the old design that are not on
this new one and what you can't tell is
that they
were also masked out because on this
current system
on this current version
there is no footprint there anyway
but if i had used that stencil without
having that tape applied
there would be blobs of solder in
different random places on here
where the old parts were
so what this has left us with is a pcb
that has solder paste on the vast
majority of the pads
horribly misaligned as you can see in
this
but at least it's on there and there are
only a couple that are missing
so then what i did was use a syringe to
apply solder paste just to the pads
that had been added in the new version
and were not existing in the old stencil
so
that was a bit of a hacky way to do it
but it actually worked out surprisingly
well now i'm being a really bad
advertisement for the system here
because i got the alignment so wrong
but
at least you can see the general concept
so
if you have a situation where you need
to make some changes to a stencil that
you've already produced
then you can possibly do that by closing
up apertures that you don't need anymore
so that was one of the issues we had to
overcome
so one of the other problems
and this is coming back to mark's
comment earlier which was
if you do not design a fuse into your
circuit the circuit will choose
a fuse for you
and
that also comes down to an issue with
something that we've seen
on these boards as we started testing
them
so
this particular board has
failed initial testing so i know that
this one is not going to do good things
and if i turn on my
handy
multimeter here
so this
on-screen display is the multimeter that
is on my bench
and the power that is coming in here
this usb cable
is actually coming from my lab power
supply
and the multimeter is measuring the
current that it's delivering
now when these boards power up what they
really should do
is pull a tiny amount of power maybe a
few milliamps
because when they initially begin the
fpga does not have anything loaded on it
there is nothing actually running
there's nothing executing
so it's a trivial amount of power draw
and this particular board one of the
things we discovered in fact the way we
tracked this problem down
is we had a couple of these boards that
failed testing but we didn't know why
and andy took
several of the boards home
and put them in a bag and transported
them home and then when he got home
opened the bag and put the pieces out on
the table
and something small fell out and there's
little silver part and i was wondering
where does that come from
and it turns out that it is the 48
megahertz
crystal
which is
where is it it's on the board
right there
it's over there
um yeah so just here you can see that
little silver part above my finger that
is the 48 megahertz crystal which is
used by the fpga
and it had fallen off
and our first thought was well there's
been some terrible assembly error like
i've simply failed to
do the reflow properly
and it's fallen off
and then we started testing some more
boards and we discovered
uh to our surprise that they were
pulling a large amount of current and
getting very hot very quickly
and demonstrating that fuse principle so
with this particular one
what i'm going to do now
is attempt a recovery i don't know if
this is going to work
but let's try it
so i'm going to turn on my lab supply
just for a moment and there might
actually be some smoke here
so let's just see what happens
watch the reading on the multimeter all
right turn it on look you can see it's
hidden amp
and
i don't see any smoke
let's see if we can arrange some smoke
one way to get smoke
now because i happen to know where this
heat is coming from is to put some flux
on there
so if i now turn the power supply back
on again it hopefully will heat up the
flux
and we will get some smoke maybe i need
to turn up the current limit
yes i think i need to turn up the
current limit let's
give it two amps and see what happens
no i think the current is being limited
maybe in the cable
but i can feel that it's getting warm
so what was happening is that when we
connected this to a laptop
to andy's laptop
it was supplying enough oh there is a
little bit of smoke coming off
yeah hang on i'll see if we can capture
that i'll stuck it stick it under the
microscope
you can no longer see the multimeter but
that doesn't matter
but the um
there for yeah you can just see a bit of
smoke coming off there
so
the 48 megahertz crystal is getting hot
enough that it is heating up that flux
that i put on it and there is a bit of
smoke coming off
yes you need to see it drifting up there
i do but i don't have an easy way of
getting it onto the stream
so it's still sitting there pulling an
amp
so
what i was thinking of doing was maybe
getting the hot air running
and
just to demonstrate that this is where
the power is going
so we saw that there was an amp of
current being pulled then
i am going to put this under the
microscope
run the hot air
and remove that
oscillator where is it right there
so there will be a little bit of noise
when i'm running this hot air
unfortunately
so effectively what was happening was
that these were self-disassembling
circuit boards they got hot enough
that it melted the solder and that was
what made the
the oscillator fall off
there's a very interesting project being
done by
that was worked on a little while ago by
someone at the hackspace to do
self-assembling pcbs
which worked on a simple like principle
you passed a lot of current through
a layer of the pcb there we go so we've
removed the crystal
it passed a lot of current through the
pcb and made it get hot so that you can
have it self-assemble
and now if i
move this back over
under the overhead camera
and we can see the current
the lab supplies currently turned off
i'll plug this in turn on the lab supply
and you can see there we're getting
about 16 milliamps being pulled by the
board
so the short is passing through the
crystal
which is one of the last things that i
would have expected to be happening
but there you go sometimes you just got
to look at where things get hot
find that
self-assigned
fuse
and then use that as a clue to guide you
to where the problem is
so the board itself still feels quite
hot to me but that's just because i've
been
putting it under the the hot air
but now if we put a new crystal onto
here
this board may well
be
uh
may well be functional so how am i going
for time
how long have i been talking so far
17 minutes
i've got ages
okay so what we could try doing is put
another crystal on this
i did not plan this far ahead
so i don't have one handy um they are in
a box over in the corner i believe
start to check that i wasn't going to
rip the microphone off something when i
walked away
so this is the box of parts
that go into
the
rockling
it's many many different parts and this
isn't all of them either many of the
really common jelly bean parts like 10k
resistors and 1k resistors and 4k sevens
and those sorts of things those are all
reels on my pick and place machine
so these are some of the unusual ones
so in here we should have a crystal
it's a microphone inverter
and a regulator power switch
no surely there is a crystal somewhere
there
not there
no perhaps i don't have it here it might
be somewhere else
so
also
sure
so let's ah let's put that aside for now
i won't proceed with adding that extra
part onto there
okay
so
the
third
uh
little area of failure that i want to
talk about
and problems that we have to resolve
is in relation to
the party button
so
this is the
add-on
which
steph piper designed
steph from elky education
and this is based on a project that she
did a i think it was a couple of years
ago
where she built a very large art
installation which was like a pedestrian
crossing
and that's why it has this shape and the
pedestrian crossing had a button on it
the regular pedestrian crossing button
but the idea was that when you walked up
to it it was a party button and you
would press it and lights would start
flashing and things would happen on this
big pedestrian crossing installation and
so what we have here
is a
miniature version
of her full-size
two plus meter high i think it is art
installation
but it's in the form of a simple add-on
that can go into a conference badge
and there are a couple of interesting
things about this design
one is that you can plug it into a badge
and it will receive its power from the
badge
and be controlled by it
so there is a transistor here which
turns on leds which will appear through
the pcb
but you can also
use this little add-on stand
which provides power
to the sayo connector
so if i plug that in and also it is
mechanical so it allows it to stand up
by itself so you can put that on the
bench and it becomes self-supporting
so if you press the pedestrian crossing
button and hold it down
you'll see that there are leds on the
back that shine through the pcb so they
are rear mounted leds
and they are changing color over time
these are leds that have built-in
drivers
that just cycle through different colors
you don't need any logic or control
system to run them
so it's a pretty cool thing
and by taking that off and applying it
to the badge
what you could do is have this turned on
by logic within the batch because it can
be controlled through these pins
now one of the difficulties that we had
with this one
is that
there were a couple of iterations of the
design
and the layout of the pcb
and the schematic at some point i think
may have diverged
we're not quite sure yet what the
the sequence was but it doesn't really
matter at this point
but what ended up happening was that we
had a design that we thought was final
based on testing that had been done on
prototypes
and then the pcbs were ordered and when
they arrived they didn't work
and it turned out that there was a
misconnection on the pcb
so
going back across to
the handy dandy microscope once again
what we can see here are the tracks on
the pcb you can see there
there are some points here for
an led there's a serviceman led option
as well
two options for transistors which is
just connected in
direct parallel so this is a dual
footprint you can either use a surface
mount part
or you can put a through-hole part on
there whichever you're more comfortable
with
and likewise these footprints are dual
footprints
it has holes in them here so that you
can put a regular through-hole resistor
and it has pads so that you can put a
surface mount resistor on it if you want
to
now
the difficulty or the change that we
eventually track down is this trace here
which comes off this connector
of that pin on the connector
right there comes down this part of the
board comes across and then it goes
through a via or via
someone needs to tell me the correct way
to pronounce that because i still don't
know after all those years they go
through that to the other side of the
pcb
it wanders up here and it comes out
and joins on the other side
up
here somewhere anyway
the issue is
that this track
goes to the wrong place so in order to
fix that
we need to do a couple of things
now i have here my handy dandy little
jewelers
grinder
so the way to apply this fix
is that first
we need to cut this track
[Music]
like that
so that there is no longer a connection
running up to the transistor and then
the next thing we want to do
is redirect that because we still need
that connection to go through
it's just that it was going to the
source
pin and we want it to go to the gate
instead
so
what
what i'd do to fix that
just change the tips on my
handy little grinder
and just buzz off
a little bit of the solar mask like that
and then we've got some exposed copper
that can be soldered onto
so then a jumper can go from that point
and it can come up and be soldered onto
this pad here
now
you might be wondering why
i would bother scraping back the solder
mask and putting a jumper on from this
point doing it right out here at the end
of this track
instead of doing it from here where
there is a nice big pad
and what i could do is run a jumper from
that point
up to here
and not have to do that fiddly work of
attaching it just on here
now the reason for this is that we want
to send these out as kits
so that people can assemble them
themselves
now if you look at one here which has
already been patched
this one has the little jumper that runs
from
this point on the pcb
and it runs up to this pad
what we could do
is solder from here but
when it comes time there are two
problems firstly when it comes time
for someone to put the header into this
point
and then to
solder it on the jumper would probably
come off
and that is if we hadn't filled that
hole anyway in putting the jumper in
place
we would need to put the jumper in place
in a way that doesn't fill the hole
and allows the
pin header to be installed
so
in planning this fix
my one of my objection my objectives was
to put the fix in place
and not have it thermally
impacted by other soldering that is
going to be done on the board after the
fix has been done
so what we're going to be doing
over the next little while
is applying this patch to all of the 50
or so of the party buttons that we have
so that when they are shipped out to
those who receive them they will get
them in the patched state
and then you can just put on the extra
parts and assemble it as you normally
would and when you put the parts on here
and solder these connections
because this solder joint is at the end
of this long thin track
the heat of doing this solder joint
will begin to flow down this track
but it's isolated enough that it should
not melt this joint at the same time
so that jumper will stay in place
even while you are soldering this header
in
so that might be a bit of a strange
design objective in terms of how that
this patch was
designed
but you just have to work with
whatever the circumstances are
um so a couple of minutes is it all
right okay in that case i think i had
better jump to some questions what
questions do people have
we're talking about whether you could
drill
ah yes okay
yes so this is referring to the solder
stencil i assume
yes
so
on this
yeah
on this pcb which
has some pads that need solder applied
to them
one of the issues is then how do you put
that solder on
the way i have been doing that is just
with a syringe
so i use the stencil to apply the
majority and then i syringe on for the
extra parts
but one thing that i was considering
was drilling extra apertures into this
pcb out onto the pcb into the stencil
i have not actually done that
and um part of that was a bit of a risk
analysis thing
because
the benefit of doing it means that it
saves me a little bit of time a very
small amount of time with the syringe
but the downside to getting it wrong is
ruining the stencil and so i did not
want to risk
putting this under the drill press
and drilling a couple of 0.8 millimeter
holes or something in it
and messing up the stencil
but modifying the stencil is something
that i would like to try in future
so once we are beyond the
mission critical aspect of this i'm
probably going to grab an old stencil
i've got a collection of stencils that
are from obsolete designs
and just try drilling and
cutting different shapes into it
and see if there is some way that i can
modify the stencil to add apertures
as well as to remove them
so
are there any other questions
could you not cut on the other side of
the board nearer to the transistor
uh quite possibly let me have a look
section okay
so could i cut up closer to there on the
other side of the pcb
yes i can't remember where that goes oh
still there
so i could cut from there
yes but
i would still need somewhere
to
well actually no that would
sorry just looking at this now
if if we jump back to this view and i'll
try zooming the microscope out so that
we can
get a better overall view of what's
going on
and turn down the brightness what's
happening is that the
the cut from here goes through this wire
which comes through here and runs up
this track to this pin on the transistor
so what we could do is cut the track on
this side and further up
but the problem is that we still need
somewhere to take a connection off this
track
and solder it onto
either one of these surface mount pads
or up here
so we need a destination for it as well
as a source for the connection
and by changing the source to this side
of the board it would actually make it
harder because we would not have easy
access to the destination which ideally
is this existing surface mount pad right
here
last question was people who only know
eagle pad do you think it's worth
switching to keycard how do you find
okay so
for the design on this this was all done
by bob
and bob is probably a good person to
have an opinion on this because he uses
he doesn't use kicad as his main
uh software he's used it for a few
projects in the past but for his regular
day-to-day contracting i think you use
altium is that right bob yeah
yeah so
for me personally i use eagle for all of
my day-to-day work bob uses ltm for his
day-to-day work
so neither of us are native kicad users
and i've used it a little
playing around with other people's
designs
i haven't made the jump into using it
i've found in i know that there has just
been a big new release that came out
only you know a week or two ago
so a lot of things like keyboard
shortcuts and things have just changed
in kicad so experiences that people have
had with kicad up until about a week ago
may not be valid anymore apparently
things have improved significantly with
that change
and
from my personal experience has been
that i found the
disjointed nature of the way the pcb and
schematic tools work to be quite
frustrating because i'm used to the
highly integrated system that eagle uses
and the idea of separating out those two
things and having them very loosely
coupled
i've i found quite frustrating when i've
been using kicad in the past
but it is certainly a very capable piece
of software
and if anyone is wanting to get started
with pcb design from a hobbyist
perspective
that is the thing that i would recommend
okay
so
are we about out of time
i think we're out of time so uh i'm
going to wrap myself up now
ah that is that is a really good point
so i am not 100 sure yet but i think
that the problem is that the pads
under the crystals or the apertures for
the crystal footprint is allowing a
little bit too so much solder paste to
go on
and it is making contact with the body
of the crystal so there is a short
circuit that is happening around the
edge of the crystal body
and
that's where the
all that heat is coming from so once the
crystal is removed there is no more
short circuit
but why that would be a problem is also
a little bit of an unexplained thing at
the moment
because the way they are connected to
the fpga
uh typically with crystals they in this
sort of format they have four pins and
two of them will be ground pins and two
of them are the pins that go to the rest
of the circuit
so
those pins are normally very very low
power they are just the
signal that comes from the fpga
which excites the crystal and makes it
oscillate so
we should be talking about
nano amps or micro amps of power through
those connections anyway
so why shorting those pins in particular
causes a dead short and very high
current drain i don't yet have an
explanation for that
okay so thank you very much we're going
to have a short break and then we're
going to be back with the next
presentation so we'll see you all soon
[Music]