foreign
thanks for joining us Andrew is a
optical astronomer
oh testing my memory
at Canton University all right so please
welcome Andrew talking about the square
kilometer array
hello everyone uh this is not the square
kilometer right this is the telescope
that I spend most of my time working on
it's the Murchison wide field array and
I gave a talk about this at my very
first Picard in 2017. this is one of 256
tiles and that tile has 16 antennas and
if a signal is coming from straight
overhead it hits those antennas at the
same time if the signal is coming from
an angle then it hits them at different
times and we combine the signals from
these antennas together with different
delays to make it more sensitive to
whatever direction we want by combining
them in analog with the right delays so
this is what uh mwo looks like this is
the escapes so this is one of 512
stations it has 256 antennas not 16 but
otherwise it's the same principle if a
signal comes from straight overhead it
hits all of those 256 antennas the same
insulin if it's coming from some other
direction in the sky it hits those
antennas at different times and by
adding those antenna signals together
with different delays we could make this
station sensitive to whatever direction
we want the difference is that with the
ski we're digitizing each of those 256
signals independently so we have a
signal that we can combine in the
computer the signal from each of those
20056 antennas with whatever delays we
want to produce a station that can be
steered entirely in software what's more
because it's in software we can steer it
in multiple different directions at the
same time we have parallel streams
applying the same addition process but
with different delays to point the solar
square but multiple different directions
in the sky you can't do that with the
dish you point a big dish in One
Direction it's pointing in that
direction with this we can point in as
many directions as we want given
sufficient computing power
but digitizing all of those signals is a
huge investment in resources you can't
do that using a computer that's out in
the desert so what we do is we take the
signals from these 2506 antennas and we
pipe the signals many many kilometers
away to a giant shielded server room
full of high-speed fpgas so each of
those 250 256 antennas has an X and A Y
polarization so that's 512 data streams
Each of which is sampled at about a
billion samples per second and then fed
through an awful lot of competing to
produce your your actual data
now I'm not involved in any of that my
job is to provide 5 volt power to those
antennas and get the data on our optical
fiber back to the computer room and in
fact I'm not involved in that either the
hardware guys designed the hardware to
do that my job is to write the python
software that talks with that Hardware
to tell it to turn the ports on and off
so that's what I'm going to be talking
about today
so the history is oh I should warn you
I'm a uni lyncher so I tend to put my
slides full of words
have them on the slides and lecture
notes when they can download but I'm
going to try not to just say everything
that's on the slide another warning I
tend to talk in like 45 minute chunks so
I really hope I don't go over time
so Kurt and uni uh over in Perth Western
Australia was given the contract to
design and build the system to produce
to provide Power to these antennas and
to get the signals on optical fibers now
the problem is you need to have almost
zero emissions this is going to be right
next to where the antennas are any
electrical noise will be picked up by
those antennas and confused with signals
from the sky
the other problem is you need to work
with very high temperatures you know
greater than 50 degree air temperatures
on site you've got to make it easy to
maintain so you don't want to have fans
that will have bearings using up things
like that
uh and you've got to make it easy to
maintain because you have an awful lot
of these things
so a few guys at Curtin uh Dave and
Brian and Philip designed the
architecture uh over a few years the
idea was you'd have a a box near the
station that would have 230 volt power
and a different group called infraris
providing that 230 volt power to us
it provides 28 ports of 48 volt DC of a
coax cable and coax cable is chosen
because it's cheaper
uh and that 48 volt power would also
carry a Communications system to each of
24 little boxes in the field and those
24 boxes with each connect to 10 or 12
antennas
and each of those little boxes would
convert the 48 volt volt power to about
5 volts and convert the electrical
signals from the antennas into Optical
signals
now inside the station we don't want to
deal with high speed stuff like ethernet
so to each of those boxes we're actually
doing a low speed serial over the over
the coaxial cable even ethernet over
optical fiber would be a problem because
that means those little boxes out
amongst the antennas have to have a
tcprp stack and some kind of high-speed
computer talk IP and we don't want that
that would produce too much RF noise
so what we use is serial and a protocol
called modbus that was invented a very
long time ago
so the system
tennis has a a short maybe 10 meter long
coaxial cable pair one carrying the X
signal on one carrying the Y and both of
them carry about 5 volts DC for the
amplifiers on the top of the antennas
and they stand about as tall as iron
they're about two meter tall Christmas
trees
the signals go from coax to their smart
boxes inside the smart boxes there are
little modules that convert the coaxial
RF like just like a signal from your TV
antenna into analog Optical
we don't digitize it we have a little
box
kind of like an SFP only it has uh two
in two inputs via coaxial cable for
analog radio frequency input from X and
my channel it has two LEDs one for the
explorerization one for the Y
polarization it puts those on the same
fiber and it modulates the LEDs exactly
according to the input voltage on the
coaxial cable so it produces an analog
light signal that turns on and off and
varies in in power exactly with the
electrical input voltage coming from
these antennas
we do that again because digitizing is
something you don't want to do because
it involves computers so we just turn it
into light pass it down on optical fiber
where it can carry for many many
kilometers without being interfered with
or or attenuated
so these smart boxes provide five volts
power from the 48 they also allow us to
monitor the power going to the antennas
and switch it off if something chooses
the cable there are lots of wildlife out
here so camels uh try and eat the grass
and they chew the cables if a cable gets
chewed it turns into a dead short and we
don't want that blowing a fuse that
someone has then go out and change so
those little boxes do things like
monitor the current a thousand times a
second if it looks like it's going into
a dead short that might blow something
up it switches it off before it can melt
any equipment
the 48 volt and comes cable to the smart
boxes go to what score will it the field
node Distribution Hub that provides the
48 volt DC provides the ability to
switch smart boxes on and off and it
provides this shared serial bus you can
talk to these boxes from out from the
outside world
so this is the Smart Box
um the box is in two bits the little
square unit you can see on the top left
with a green cable attached to it that
is where all of the Active Electronics
is that power supply the microcontroller
the serial Communications bit all of
that is in that box that box is entirely
sealed both physically to you know
eliminate water and ants getting in and
electrically to stop radio frequency
noise getting out
that box then Clips onto the underside
of this environmental package and the
environmental package the big flat bit
that doesn't have anything active in it
it has a press button it has some
switches to tell it what its address is
it has the analog components that don't
have any high speed on off uh switching
that might generate noise it only has
analog stuff so it needs to be shielded
from the elements but it doesn't need to
be screened electrically nothing it has
inside can radiate outwards to interfere
with the with the antenna signal
on each side of the box there are uh six
dual polarization inputs X and Y and you
can see that the coax connectors there
and
looking at the inside of the Box this is
from the top down on the left hand side
you can see here the zero two those are
little jumpers sitting in um eight pin
sockets and that defines what the
address of this thing is modbus has a
concept of an 8-bit address to Define
what physical thing you're talking to we
didn't want to put the address in the
electronics package in the firmware
because that means that if somebody
comes along and swaps it out then they
would need to set it to the right
address something needs to know what
antennas are physically connected to
that box the antennas are physically
connected to this package here so we
wanted to put the address the bit that
specifies the address in this package of
that stayed physically connected to the
mesh and the antennas
so if you come out you install this
thing you say it's Smart Box number two
you note that you're connecting antennas
you know one through 15 through 20 to
this box that then lives there forever
and any changes that happen particularly
for changing out the electronics module
you swap in a new computer module it's
completely stateless it doesn't need to
know anything about what it is or where
it is it just gets told when it reads
those switches that it's address number
two
so the um
the bit that actually turns the radio
frequency into Optical are those silver
boxes
on the right there's a close-up of a few
of them they just take a couple of thin
cables in carrying the electrical radio
signal from the antenna and they have a
single fiber going out
um to carry the analog radio frequency
data
this is all aggregated together into a
12 fiber ribbon that then goes out
through a hole in the box and back to
the the station to be aggregated in into
a bigger Collective to go back to the
main site
the electronics package
has three boards and some filters this
is upside down so the d-connectors that
that plug up into the the Box are on the
bottom there
so each of those little D connectors has
a filter board
because those d-connectors only carry DC
or LEDs that can be switched on and off
for example so there's filters make sure
that when you switch an LED on it
switches on in a millisecond not at a
micro second the rise time is slow
enough that it doesn't generate noise
we've got a microcontroller board with a
pic24 microcontroller in there uh
surrounded by a metal box to keep it
shielded then we've got the what we call
the pdoc power and data over coax board
that wraps the the serial Communications
on top of the 48 volt DC
and we've got a power supply which is a
it's basically a switch mode power
supply but it's designed to be
incredibly low noise to convert the 48
volts down to about 5 volts it's
actually about 4.7
and all of this works up to about 85
degrees Centigrade
there it is uh look that looking down
from the top
and also has this ndh the field node
Distribution Hub Electronics package so
you can see a Smart Box sitting on top
of it there the Box itself has 48 uh
sorry yeah 28 ports the 28 ports and you
can't see it on there the 28 bullets of
those coax connectors and again we use
those coax connectors because they're
cheap they're cable TV F connectors you
crimp it on the cables cheap the
connectors are cheap uh you have no idea
how expensive DC connectors can be that
are rated for outside you can pay
hundreds of dollars for a single
multi-pin connector
uh and those boxes those ports are
designed to be redundant we only need 24
boxes we have 28 ports so a technician
can just you know swap it to a port
that's working if one of them is broken
there's another package that Aggregates
the fiber I won't go into that because
that's just Plumbing there's no Active
Components in there
inside that box we have the power supply
module on the right uh that's a seal box
that has our 240 volt to 48 volt power
supplies it also produces 12 and DC for
12 volts and 5 volts DC for the rest of
the electronics
we've got the communications box down
the bottom that goes from fiber to
ethernet and then from ethernet to loose
bin cereal and on the left hand side on
Port 3 on the number three we have all
of the the boards that provide the
outputs to the to the actual smart boxes
so this is a Communications Gateway we
use an off the fiber mid off the shelf
media converter a very short little
length of coax going to a whiz note
um ethernet to serial converter module
and then a filter board to make sure
that even that serial has low rise and
fall times so that it can be transmitted
be carried outside the box without
worrying about RFI
and a pair of fibers comes in from the
outside world to provide Communications
to the control room
that board there where it's actually a
bunch of boards we've got 28 what we
call p-dog boards they're the exact
equivalent of the PW board in the Smart
Box they're the other end of the link
they insert the 48 watt power and the
serial over the top
and provider the main board there
provides this shared serial bus between
all 28 ports and allows you to turn
ports on and off again if a camera to
use the cable you want to turn it off
it actually has two microcontrollers on
it the same microcontrol that's in the
spot box one of them handles the
peripherals turning ports on and off we
ran out of ports on that one so we have
another one that handles three uh elon's
three serial ports one of them is the
outside world to the West End module one
of them is the the fnpc on this board
that handles all of the ports that go on
and off and the third one is a
multi-drop Serial bus that goes to the
actual smart boxes
modbus uh was designed in
nine uh called modbus because it was by
a company called modicon
it's a client server protocol uh the
idea is that the server doesn't have an
address clients do each client has an
address from 1 to 255 with zero is
broadcast there's lots of variants now
all of them have the same packet
structure that was originally set down
in 1979 but they frame it in different
ways there's modbox TCP where it's
encapsulated in a TCP packet that's my
buzz rtu which is raw binary on a Serial
line because my bus asking you there's a
whole bunch of variants but the idea is
they all have the same packet structure
with different ways to signify start and
end of a packet
so a transaction is a command from the
server and a reply from the client
clients can't initiate anything they can
only respond when sent packet so the
server has to continually pull all of
the smart boxes and the fnpc to find out
what's going on
the basic structure is the client
address which is one byte followed by
the command code what you want to do
followed by the data for that command
followed by checksum
and the reply is the client address
saying hi I'm replying from the address
you to send that message to
the command code which is hi this is the
message the command that I'm replying to
and whatever the data is usually the
contents of registers and it checks up
the client always replies with its own
address so that the server can make sure
that it's not getting out of sync if
there's an error if the command didn't
ask for example then the client sends
back the same structure only with a
command code uh with added with plus
128. so the high bit set that means that
command did not work
and an error code called a modbus
exception code there's a bunch of those
and checks up
we're using the ASCII version of modbus
now that means we're actually sending
hexadecimal ASCII characters so we put a
column at the start of a packet
Carriage Eternal Line Feed for the end
of a packet and we send for example the
letter A followed by the digit 0 as 2
8-bit ASCII values to denote one byte
meaning a zero and a register for
example
and it's got a single byte checksum
there's two's complement of the the sum
of all the values that's a terrible
checksum but you know that's in the
standard so that's what had to use we
wanted to keep it compatible with
industry
um uh libraries and tools and stuff like
that
we're using a subset of modbus commands
that handle something called holding
registers as modbus handles a whole heap
of stuff including writing to individual
relay elements with single bit commands
and stuff like that we're just using a
bunch of 16-bit registers so we're using
command codes three which means read
some registers six which means write one
register and 10 which means write a
bunch of registers
so for example if you want to read two
registers from Smart Box one starting at
register 11 you'd send address one
uh
which would be column to start the
packet and then zero one meaning Smart
Box one you'd send then the zero three
which is the read registers command zero
three you'd then send zero zero zero a
for the register address uh 11 minus one
you subtract one to put in the packet
for some reason and then you'd send the
number of registers zero zero zero two
and then you send the checks up and then
you send a carriage return on a line
feed so it's just a print statement to a
device in um a write statement to a
device in Python
and the Smart Box would send back
something similar colon zero one zero
three and then the contents of those
registers
so the Smart Box and the fnpc just have
a bunch of 16-bit registers you write
values to them if you want to change
those registers you read values back if
you want to
um see what they are modbus doesn't
specify anything about those registers
it's up to the individual application to
decide what registers mean what
so while the development guys while the
hardware guys
decided on python code to prototype and
test all of this
so first of all I wrote a transport
library to handle the serial
communication using pi serial and all
the modbus packet assembly and decoding
and that's used by both the server and
some simulated clients
I wrote some controller classes that
represent the internal state of the
remote Hardware it's just a class called
smartbox there's a bunch of attributes
that Define you know the 48 volt
incoming voltage and the current on Port
3 and things like that and each of them
is represented by a register and there's
a couple of functions to read the
contents of those and populate the
attributes in your class instance and
those functions to you know write back
if you change the the port configuration
it writes it back to the device
but I also read simulator classes to act
as clients they respond to modbus
commands by reading and writing their
own internal register States but they
also simulate random mock Telemetry
they'll sit there and slowly change
their internal temperature and current
on Port 3 and stuff like that over some
random slowly varying cycle
so that we can you know simulate some
real hardware and actually see changing
numbers coming back
so the transport Library defines a
connection class but I wanted it to
handle this shared serial bus that
happens at the at a station so in the
station you've got one serial Link in to
a device but there's a whole bunch of
things you can talk to Via modbus on
that one link
so I created this class that has a
connection instance and then you can
share between threads and the idea is
that each simulator object whether it's
a simulated Smart Box or a simulated FN
PC would be a thread it would be a one
or actually two threads that share the
same connection since all of the other
simulated smart boxes so each thread
gets its own independent input buffer
any data received from the physical
serial Port is written to the input
buffer from every thread every thread
then consumes it as needed if it senses
if it detects that something's talking
to it
so if any thread where I started for
connection instance that data is only
written to the external serial Port
that's because that's how the physical
Hardware works in the in the station uh
if one Smart Box writes some bytes of
data only the the outside world sees
that via the the communications Gateway
the other smart boxes don't see that
which helps to keep it more reliable so
it's not a full multi-drop bus it's half
a multi-drop bus
so the controller classes get past an
instant of a connection object and a
modbus address on creation that the
device that they're actually going to
talk to the attributes hold this
Telemetry data and we've got methods to
read it from the hardware and push it to
the push the current contents the
attributes to the to the hardware
and then there's a station control class
that wraps all of those so this is
written to control an entire field of
256 antenna 24 smart boxes and fnzh and
it just creates a bunch of controller
classes that can create 24 Smart Box
classes it creates uh one physical one
fndh class so it's got methods to
initialize and control the entire
station turning the ports on you know at
smartbox three are you there yes if you
are then I'll create a control class for
you and runs an entire station so it
kind of wraps it wraps all those
individual classes
the similar classes get past an instance
of a connection object and a mod match
address on creation just like a
controller but each of them runs two
threads one of them is a lesson thread
that's just responding to serial data
that it sees on its input buffer and
saying is that addressed to me no I'll
ignore it yes I'll respond and I'll give
it whatever it wants I'll give it the
register addresses that it's asked for
or I will write the data that I'm given
to my an internal state
and the other one is a simulator Loop
that just runs in parallel and slowly
randomly drifts in temperature and
handles all of the state the firmware
does on a real device if the temperature
gets too high go into an alarm State
turn all the thoughts off if the current
on Port 3 is too high turn that Port off
because the camel has eaten the cable
so that sits there and stimulates a real
box and the list of loop acts as the
communications to to talk to the monitor
Control software
so I've also got a
so the monitor control side starts out
by talking to just an fndh and it says
all right you're initialized now turn on
Port one and when the simulator the
station's email addresses are Port one
has been turned on but one I've been
told is connected to smartbox three I
will fire up an instance of a Smart Box
simulator start the two threads for that
class to simulate a physical Smart Box
bind it to modbus address number three
and now the monitor control system can
talk to Smart Bus 3.
so as each Port is turned on more of
these Smart Box simulator classes are
created more of the threads are started
to simulate their existence and this
station simulator can pretend to be an
entire station
up with up to about 50 threads
so if a port is turned off it kills
those threads deletes the Smart Box
instance and that's Mark boxes now off
so by early 2021 before we even had
Hardware ready to test I had a full ska
station ready in software
so I had a station controller that could
control an fndh and 24 smart boxes I had
an ethernet cable going to one of these
little whiz net modules sitting on my
desk I then had a rs232 Serial cable
going to another computer which was
running the sessions emulator so that
simulated an fndh and up to 24 smart
boxes that can be turned on and off
and that's how he developed the hardware
we had a full station simulator and we
could simulate anything from an
individual Smart Box all the way through
a full set of them so we could do things
like what bandwidth do we need to
actually monitor these things at any
kind of reasonable rate how much dead
time do we have on that for doing other
stuff
how do we do things like handle
servicing if somebody wants to service a
box we don't want to take down the
entire station how do we handle you know
responding to a press button on a box to
have the the monitor Control software
turn off just that box so that a
technician can work on it so we we work
through designing uh stuff like how the
the maintenance would work as well as
verifying the actual firmware so that
the guys doing the hardware and the guy
doing the firmware we're working in
parallel with the simulator codes so
that they could validate the um the
firmware Behavior against the the
simulators and the simulators changed as
it turned out that some things went
possible to learn Hardware obviously the
code had to change when the firmware guy
developed new ideas like the ability to
push remote firmware and do cool things
the stuff I had to change so from 2021
through to basically now it was a very
responsive
um three-way design process
so the current status is that we've had
a prototype running in the desert for
about 18 months it only has four smart
boxes connected to it but it's otherwise
a full working system
we have built and delivered a full
system of 24 smart boxes and an fndh in
all of their respective enclosures and
fiber set up to
Geraldton and it being installed on the
field last week and it's powered up and
being tested remotely and that will be
part of a new SK prototype station
called avs3
we've handed off the entire design our
hardware and firmware for sklo
construction it's been taken up by a
manufacturer for mass production
and the sko guys are riding their own
Control software they have a very
controlled development process I'm not
part of that control development process
so they're writing their own software to
actually run it but it's also based in
Python it's using a system called Tango
controls which is a a framework for
interacting with Hardware control
systems
how am I doing for time running a bit
short
so my code is on GitHub it's MIT
licensed
the uh Kyra The Institute where I work
is there mwa telescope which is my baby
um Tango controls if you're interested
in that and the little wiznet modules
are pretty cool
and this animation here is just some
tech drawings showing how the bits all
fit together
and I promised
a live view of some telemetry
which I'll show you now
so this here is
live data from Smart Box one
so you can see there
around all this by the way I wrote a
rapper that actually you know grabs the
data puts in a database writes it to
carbon whisker a whisper around Robin
databases and grafana to view it so that
low level stuff was the guts and there's
wrappers around the outside of that so
that's the live data it's about
20 degrees out there this is over the
last 30 days you can see the temperature
is going up and down
the individual currents in milliamps for
each of the
fems on smartbox one
and you can see a whole bunch of stuff
and we've got a bunch of different
grafana pages for digging into this and
we use that to do things like working
out temperature dependence uh how well
they handled high temperatures on site
during summer things like that
and that is it from me
[Applause]
thank you very much Andrew
now I learned my lesson earlier and not
relying on my memory this time
we're going to take one question from
Discord and then one from the room so
signal to Jack if you're in the room and
you have a question because he's going
to be the one running the mic to you
question from Discord is how do you
correct for attenuation in the fiber is
there a pre-calibration or regular
recalibration and similar temperature
stability of lead output yeah that style
done by the science guys they look at an
object in the sky of a known brightness
and they use that to calibrate each and
every antenna sensitivity and they would
be doing that either every day or before
every observation that's something you
have to do for any radio telescope and
they also calibrate the length of the
cable because when you're measuring
delays the length of each individual
cable is very important so they can also
determine the length of each cable all
the way from the antenna all the way out
to the correlator rim to the nearest few
millimeters by looking at known objects
in the sky
excellent talk
my question is a little more General as
someone who was not comfortable with
control systems in college
like what what's the I mean for someone
uh for generic people why modbus why not
can bus or other application layers over
TCP
I'm not sure we wanted something that
would be suitable for a low speed
microcontroller and we wanted something
where we didn't have to have
any kind of smarts on site we wanted all
of the intelligence to be in software on
a Linux server in a control room
otherwise we'd have to have you know
firmware in the station box the fndh
talking to each of the individual units
and then aggregating it and then sending
it back by some other protocol we wanted
to keep it consistent but as to why
modbus was chosen or anything over
anything else that wasn't my decision
that was um Dave and Phil and the design
guys
thank you again Andrew and thank you
from the conference for giving a talk
there's a round of reports please
[Music]
[Applause]