Monday, January 13, 2014

philam UI

A little less work post-Christmas on philam (the current name of my Flex 6700 client), so far.
Nevertheless I've tidied up the code a little, particularly in terms of how streams from the radio can be associated with stream handlers.

One of the the truly nice things about Haskell is how elegantly concurrency and parallelism are integrated.  Indeed as a pure, functional language Haskell lacks so many of the features that cause real problems with concurrency in other languages (e.g. destructive assignment, shared state, uncontrolled side-effects).  While Haskell has many tools for concurrency, even the basic explicit concurrency (i.e. spawning a thread) is simpler and less burdened with issues.  Haskell threads are lightweight 'green' threads that are scheduled onto hardware threads by the Haskell runtime.  That means you can create hundreds or thousands, even tens of thousands of threads without gumming up the works.  Moreover, Haskell threads behave very nicely when they are doing IO, they can be terminated without any problems and tasks like IO can be timed-out by just wrapping a 'timeout' function around some action.  All of this lends itself beautifully to asynchronous handlers of the kind needed to react to data stream packets and notifications.

Philam's comms module now has a "startInStreamHandler" with the following type:
startInStreamHandler :: FlexRadioConnection -> InStreamSpec -> PacketHandler -> Maybe PortNumber -> 
IO (Maybe FlexInStream) 

This takes:
A radio connection, to identify which radio the stream will be from
A stream specification, to describe what kind of stream is required
A packet handler, a function to handle each packet of the stream
An optional port number, provided if a port other than the standard VRT port is to be used

If the stream could be created and the handler was successfully launched, then the function returns a "FlexInStream", which is the record that keeps details of the created stream (such as the stream ID used to identify it in the radio).

startInStreamHandler forks a new Haskell thread to handle incoming packets from either the standard port (on which multiple streams are multiplexed) or the 'private' port.
If the common VRT port is used, then startInStreamHandler will also create a queue onto which the shared standard port listener for the VRT port will demux and dispatch packets of the stream.  Haskell's software transactional memory (STM) package has a TQueue data type that works very nicely for this, whose 'read' function blocks if there are no elements in the queue, providing the synchronisation needed to do work as soon as data (i.e. a new packet) is available.

Although it looks like a 'packet handler' would lack context and therefore be unable to do anything useful, Haskell's closures provide all the context you need for arbitrary context depending on purpose.   The audio stream decoder featured in my prior post has now been factored into a PacketHandler, consisting of a literal packet handling function and a cleanup function:

-- | A handler for processing of an audio stream            
audioPacketHandler :: IO PacketHandler
audioPacketHandler = do
    !samples <- newEmptyMVar
    startStreamEngine 24000 1000 1 samples
        audioPacketHandler' :: VRTIFPacket -> IO ()
        audioPacketHandler' packet =
            case packet of
                VRTIFDataPacket{..} -> do
                    -- Current stream payload is alternate L-R stereo pairs of IEEE-754 32-bit floating point samples.
                    -- These are delivered at 24000 L-R samples/s 
                    -- We need to read Word32 samples from the packet, convert these to a float, average them and then 
                    -- convert these again to the dynamic range of an Int16 for sending to OpenAL
                    let decodeStereoPairsToMonoInt16 :: Get Int16
                        decodeStereoPairsToMonoInt16 = do
                            left <- getWord32be >>= return . wordToFloat
                            right <- getWord32be >>= return . wordToFloat
                            let mono = round $ (left + right / 2) * fromIntegral (maxBound :: Int16)
                            return mono
                        allPairs = whileM (fmap not isEmpty) decodeStereoPairsToMonoInt16
                        (!result, _) = runGet allPairs vd_data
                    case result of
                        Left !err -> putStrLn $ "Error decoding audio data: " ++ err
                        Right !monoSamples -> putMVar samples monoSamples  -- Send to audio engine
                VRTIFContextPacket{..} ->
                    -- Error, we are not expecting a context packet here
                    errorM logComms $ "Context packet not expected in audio stream"   
        audioCleanup :: IO ()
        audioCleanup = 
            -- Shut down the audio engine
            putMVar samples []  -- Send empty samples list to engine
    return $ PacketHandler audioPacketHandler' audioCleanup

The function returns an action (signified by the IO) which contains the twin aspects of handling a single packet at a time and also cleaning up (the audioPacketHandler' and audioCleanup respectively).  Both of these functions are defined in the context of the outer audioPacketHandler scope, which runs a sequence of actions to set up context before returning the PacketHandler action.  In this case, these actions initialise the audio engine with its synchonisation object for sending samples.  This is part of the closure of both PacketHandler functions, allowing them to utilise the context and tear it down properly when finished in the cleanup function.

The queue handler code, called by startInStreamHandler in the case that the stream is to be delivered on the shared standard VRT port does the following:
-- | Run a packet handler on a queue   
runQueueHandler :: FlexRadioConnection -> FlexHex -> PacketHandler -> TQueue VRTIFPacket -> IO ThreadId                                     
runQueueHandler connection radioStreamID PacketHandler{..} queue = do 
    -- The termination action
    let onClose = do
        noticeM logComms $ "Terminating handler for stream " ++ show radioStreamID
        -- Terminate the stream in the radio
        terminateStream connection radioStreamID
        -- Perform packet handler cleanup
    -- Launch handler 
    (flip forkFinally) (\_ -> onClose) $ forever $ do
        !packet <- atomically $ readTQueue queue

        ph_handler packet               

The handler uses 'forkFinally' to launch a 'forever' action that will continue to get an item from the packet queue, whenever one becomes available, blocking otherwise.  The packet handler is asked to deal with each packet when one arrives.   An onClose function is defined which will terminate the stream if anything happens to it that causes this concurrent routine to terminate.  This function calls terminateStream which does common termination actions, such as removing the stream record from the connection record, then gets the packet handler to clean itself up.

Besides this improvement to the queue handling, my attention has turned to the UI, per my comments in the last post.

I have again surveyed the Haskell GUI-binding landscape and on balance I have decided to stick with gtk, the one cross-platform toolkit I have used before.  I had wondered about wxWidgets (and the wxHaskell binding), but the Haskell binding hasn't been updated for a while.  Conversely, a gtk3 binding has literally just been uploaded to the Haskell package repository Hackage.  Not only does that bring Haskell bang-up-to-date with the gtk framework, but there is also an additional "gtk3-mac-integration" package that will make the UI work well on a Mac by using the Quartz backend for gtk3.

So, having decided to stick with gtk I had to build the underlying gtk libraries on which the Haskell bindings are based.  I had previously built gtk2, so the usual build adventure lay ahead on me.

Indeed, things were a bit awkward to get going and it took a few hours to get through all the gotchas.

In the end, this is what is required:

  1. Get ready for building the Haskell binding:
    cabal install gtk2hs-buildtools
  2. Install the gtk+3 base with homebrew:
    brew install gtk+3
    This will finish by printing a caveat about dbus, which must be followed now or later
  3. Ensure you have libxml2 with the python bindings (I had it without, so...)
    brew reinstall libxml2 --with-python
    This will finish by printing a caveat about setting the PYTHONPATH, which must be used
  4. Ensure you have gtk-doc installed
    brew install gtk-doc
  5. In order to build the Haskell bindings, you must make config visible:
    export PKG_CONFIG_PATH=/usr/local/lib/pkgconfig
  6. Unfortunately on Mavericks there are issues with clang, so gcc must be obtained:
    brew tap homebrew/versions
    brew install gcc48
  7. Cairo can be tricky, so better try the Haskell binding for it first:
    cabal install cairo --with-gcc=gcc-4.8
  8. OK, now we can compile the read of the GTK 3 binding:
    cabal install gtk3 --with-gcc=gcc-4.8
  9. Now we'll need the latest glade UI builder.  So, download the latest glade sources and unpack the tarball.  Glade is tricky to build clean.  You have to do the following just to get configured:
    brew sh 
    export PKG_CONFIG_PATH=/usr/local/lib/pkgconfig
    export CC=gcc-4.8
    export PYTHONPATH=/usr/local/opt/libxml2/lib/python2.7/site-packages:$PYTHONPATH
    The 'brew sh' creates a shell with all the homebrew environment included.
    We set gcc-4.8 as the compiler to use.
    We set the PYTHONPATH, per the caveat from homebrew (see above).
  10. Now we can actually compile and install glade:
    sudo make install
  11. Now you have to do the LaunchAgent configuration per the other homebrew caveat, if you haven't done this already (i.e. copy the agent plist, register it and start it).
  12. Even now, you are not done.  Glade will not yet run because it doesn't have compiled "schemas":
    glib-compile-schmas /usr/local/share/glib-2.0/schemas
  13. OK, maybe 13 is lucky for some.  We can now finally run glade:
  14. \o/

Ugh.  So I'm now ready to start hacking on the philam UI!

Friday, January 3, 2014

Learning to drive the Flex 6700 programmatically

I now have several days of hacking under my belt and have made some nice progress in learning the ropes and getting the radio to use useful things.

As I intimated previously, I am really impressed with what Flex have built (...and indeed are continuing to build).  The 6x00 is a unique radio and it has been very much fun indeed to get acquainted with the programmatic interface of the radio as an appliance over the ethernet as well as enjoying operating it with the SmartSDR Windows software.

My initial goals have been to get a basic grasp of the control functions and data streams over ethernet.  Flex have a nicely designed set of control sentences that are constructed and sent to the radio once you are connected.  Before connecting, you can discover radios on your LAN via a periodic broadcast  that each radio makes once it is initialized and ready for client connections.

To facilitate the usual business of sending command and reconciling responses with those commands I have built a blocking command queue in Haskell.  This handles boring stuff like allocating monotonically ascending message IDs and also simplifies internal addressing of radios so that any connected radio is henceforth addressed by serial number.  Incoming traffic (which can be command responses as well as asynchronous status messages) are immediately delivered into different queues from which various handlers can subscribe to consume them.  Calls to send commands to the radio are handled through a central dispatcher that blocks on the specific response becoming available in the responses queue.  Haskell makes this sort of thing rather easy and completely thread-safe through the use of the STM monad, which implements software transitional memory.  The STM library has a range of useful concurrency tools, amongst which is the TQueue (transactional queue) that I use for all the received message queuing.

Haskell also handles threading beautifully - and it's fast and very flexible.  It runs its own scheduler that allocates code to be executed on available hardware threads.  Even if you restrict Haskell to a single thread, it will still do a great job of cooperatively multitasking between concurrent routines in your program... even when they are doing IO.

Importantly, Haskell is also cross-platform (at least for most platforms people care about: Mac, Windows, Linux).  The core of Haskell runs on many other processors and platforms (such as ARM/ Raspberry Pi) but of course that doesn't mean that all the libraries I need would work beyond the core desktop platforms.

Once the command dispatch mechanism was working I added some convenience wrappers around a number of the available command sentences, including:
  • Getting version numbers (a simple initial command to test things are working)
  • Querying for active receiver slices
  • Creating and deleting receiver slices
  • Tuning a receiver slice
  • Requesting an audio stream (from a slice)

An early objective of my experimentation was to get streaming audio to be played on my computer and in preparation for that I have had to survey the available cross-platform audio libraries.  I've had experience with Apple's excellent CoreAudio before, but I've never tried to use a cross-platform library.  It turns out there were several to choose from: PortAudio, FxModRaw, PulseAudio, OpenAL and a few others I didn't recognize.  The main feature I needed was a 'simple' streaming PCM playback and for the future also audio-in.  In practice things are a little messy with these libraries with several of them being reliant on versions of a binary dependency that I found difficult to compile on the Mac at least ("difficult" meaning having to hack build files to get things working).  The OpenAL library, despite its age - or perhaps, because of it - was very easy to get set up.  The only problem with OpenAL is that it isn't really a streaming engine per se.  Rather, it provides a basic mechanism to queue buffers sequentially for playback.  It is left as an exercise for the consumer to set up whatever higher-level sample management mechanism they may deem appropriate.  Because of this, I had to set about writing a buffer management component that asynchronously accepts samples from a source and queues filled buffers to playback (retrieving used buffers once OpenAL has finished with them).

Once this little audio engine was completed I turned my attention to the VITA stream format.  Unsurprisingly, there is no off-the-shelf decoder for VITA-49 standard streaming in the Haskell package repository 'hackage'.  This has meant rolling my own from specifications that I have found online.  Haskell has some great tools for marshalling binary data too.  Specifically, the ByteString and Binary packages provide high-performance processing of data packets/streams.  The Get and BitGet monads in the Binary package make building bitwise decoders very easy.  As well as the VITA packet decoding, I have used a Get decoder to prepare the audio sample stream into an appropriate format for OpenAL (which currently only supports int16 samples).  Here's the code:

case decodeVRTIFPacket msgBytes of
    Left err -> errorM logComms $ "Could not decode VRT IF packet because: " ++ err
    Right VRTIFPacket{..} -> do
        -- Current stream payload is alternate L-R stereo pairs of IEEE-754 32-bit floating point samples.
        -- These are delivered at 24000 L-R samples/s 
-- We need to read Word32 samples from the packet, convert these to a float, average them and then 
-- convert these again to the dynamic range of an Int16 for sending to OpenAL
let decodeStereoPairsToMonoInt16 :: Get Int16
            decodeStereoPairsToMonoInt16 = do
               left <- getWord32be >>= return . wordToFloat
                right <- getWord32be >>= return . wordToFloat
                let mono = round $ (left + right / 2) * fromIntegral (maxBound :: Int16)
                return mono
            allPairs = whileM (fmap not isEmpty) decodeStereoPairsToMonoInt16
        (!result, _) = runGet allPairs vp_data
        case result of
           Left !err -> putStrLn $ "Error decoding audio data: " ++ err
           Right !monoSamples -> putMVar samples monoSamples -- Send to audio engine

Even if you are not familiar with Haskell, you can probably grok was is going on here.
A VRT (VITA-49) packet is decoded and if that succeeds then another decoder is run on its "vp_data" component.  This decoder is mostly composed of the "decodeStereoPairsToMonoInt16" function, which lives in the Get monad as you can see from the "Get Int16" return type.  That function only handles a single left-right pair of samples - by reading them from the byte stream as 32 bit words in network order and converting each to a Haskell Float.  These two values are then averaged, scaled to the dynamic range of an Int16 and then returned as a single mono sample.  To perform this treatment on all the bytes in the stream ("allPairs"), the monadic loop function 'whileM' is used to apply this decoder to the stream until it is "not isEmpty" (i.e. empty).  The runGet function actually performs the Get action on the stream, returning the result (the stream of mono samples if all is well).  Assuming the mono samples are indeed returned, then these are handed to the sound engine by filling an 'MVar' with "putMVar samples mono samples".  MVars are another Haskell concurrency feature that implement a kind of threadsafe pigeon-hole where values can be 'put' and 'taken' by different threads with put blocking if the pigeon-hole is still full and take blocking if it is still empty.  The buffer-filler of the audio engine uses this mechanism to asynchronously take any number of proffered samples and ready these for playback.  

With the basics of control and audio working I will now turn my attention to some UI.
I will have a similar question as to which cross-platform UI library to use.  In previous experiments I have looked at GTK and wxHaskell.  Most of my actual dev time on Haskell GUI projects has been with GTK.  This works OK on a Mac with X as a dependency... although in theory you can build a gtk library that works directly against Cocoa and therefore removes that issue.  I have no idea whether GTK works nicely on Windows at this time.  wxHaskell might be a better choice on this occasion, so I'll spend a little time trying to figure out where that library stands (the underlying wxWidgets have just been updated to v3).

Sunday, December 29, 2013

API access, more testing, lots of fun

Long time, no post (... real life intrudes on my fun, unfortunately).

However, recently Flex have been kind enough to grant me access to their API program, so I have been learning and experimenting with connecting to the radio over ethernet.

This is every bit as fun as I had hoped, not least because it brings together some of my biggest passions: radio, digital comms and functional programming (the latter because I'm creating a Flex 6x00 Haskell API as I go).

Visibility into what Flex have done in regard to the radio as an internet appliance further reinforces my positive impression of this product and the company.  Clearly, the 6x00 is a product resulting from a remarkable amount of combined intellectual capital.  It continues to be very exciting to contemplate the opportunities afforded by bringing together the technologies in the 6x00.  Without a doubt this was a very ambitious project, especially for such a small company, but I really respect what they have accomplished already here, especially 'in the box'.

As an aside, it's clear that the SmartSDR client has a long way to go to fill out features and ergonomics, however even there I would commend Flex for making a great start - the UX is clean and uncluttered, emphasizing the signal visualization in the panadapter(s).  SmartSDR does not semi-permanently surface every single control a la HF radio front panel, which I think is a big mistake of many SDR interfaces.  Doing a 'genuinely digital', simple, yet efficient UI for an SDR is far more intellectually challenging than just sticking controls everywhere on the screen (even if they are organized in different panels).  There is of course always a 'matter of taste' involved in UX and naturally also a large degree of hysteresis due to experiences that users become habituated with.  In Amateur Radio it's certainly true that many operators prefer hardware front panels with many options immediately available via dedicated controls.  However, I feel that the default views for software controls should be clean as possible, even if there are options for pinning additional controls in the UI so they are available with fewer clicks or keyboard interactions.  One thing it's hard to argue with concerning SmartSDR today is that the basic panadater view is awesome, affording such an amazing real-time, high-fidelity view of the received signals.  It's wonderful to have a realtime frequency analyzer as your view on the world (and from experience with my son, it's also an amazing learning tool). That's not unique to SmartSDR of course, but the emphasis and fidelity of this feature is the best I've personally seen.

Back to the API, unsurprisingly my initial goals are to get signal data streaming from the radio.  Hopefully I can send audio out without too much trouble and then move onto some of my ideas for scanning the band and deducing signal locations and modes.

Finally, Adam has run a fresh barrage of tests on my 6700 with the possibility of doing even more.  The reasons for the number of test runs are twofold:

  1. The radio is evolving with features and fixes that have a bearing on the testing
  2. There's a lively discussion ongoing regarding appropriate and effective metrics for SDRs given their entirely different modus operandi compared to analog radios.   This discussion is converging on some methodologies and metrics that should be a better fit for meaningful comparisons across SDRs (in particular) and additional test runs are collecting relevant data.  

Saturday, October 5, 2013

Another test session with Adam Farson, putting V1 under the microscope

I spent another five hours, thereabouts, at Adam's place with the Flex 6700 on the bench.

We did a full set of NPR tests, IMD tests and some quick TX power and phase noise tests.
We ran out of time for further TX tests, so another session is scheduled for next weekend to complete those.

Adam will of course be preparing and releasing a report in due course, now that the radio has reached V1.

Monday, September 30, 2013

V1 is out!

Flex Radio put V1 of the Flex 6x00 SmartSDR and firmware up today, so the radio is now available for purchase to non-preview customers and fully supported in its intended initial form.

I have a new date at the weekend to drop the radio around to Adam's place again for some new testing.
It's open season to see just how the 6700 will fare - and this time we'll be publishing the results.

I'm sure the guys at Flex will be trying do get some much needed post-push rest, although the business of commercially supplying and supporting the radio has just started in earnest!

I still haven't heard from Flex Radio regarding the SDK/API.  That's probably because they've been completely swamped in getting V1 out... but I really hope to hear something soon about this!

Friday, September 27, 2013

v0.16.4 lookin' good

0.16.4 was released this week and seems to demonstrate that Flex are nicely closing on their goal of a solid V1 in a few days.

Aside from simple UI bugs, there were clearly a few important issues with the prior builds.  One of the most obvious for me had been the ineffectiveness of the noise DSP features.  With 0.16.4 these seem to be functioning as I'd expect in terms of actually having the expected effect.

The radio is a pleasure to use and really seems to pull signals off the air.  Once V1 has dropped, I shall attempt to perform an A-B comparison with the IC-7800.  That will also be the time for another test session with Adam Farson.

SmartSDR is very easy to use, if still a little vanilla in terms of features and convenience options.  For instance, the obvious dragging of a receiver slice to tune it has no snapping feature.  So you typically have to drag then twiddle the scroll wheel on the mouse a little (which does have a snap) to get the 'whole number' frequency that the transmitting station is invariably using.

Of course, we're not getting a number of fairly 'basic' radio features until January, with a completion of the originally specified feature set delivered through the balance of 2014 - but Flex appear to be doing the Right Thing - consolidating the core functionality to an acceptably high quality before building out new features.

Friday, August 30, 2013

A little wait for the next preview release (and the next NPR tests)

The next preview release of the SmartSDR and 6x00 firmware was due out about now, but apparently this will be a little delayed now due to some quality issues detected in this cycle.

Flex operates several phases of development and testing.  Each cycle will produce builds that are sent to 'alpha' testers, then to a wider set of 'beta' testers if a build looks solid.  Finally, if the beta testers are happy then a good build will be released as the next preview release.  Thus, preview releases are supposed to be somewhat stable and usable (modulo a set of supported features).

While I'm chomping at the bit to test the next version, especially if it has the signal path optimisations we've been promised since our initial testing, I'm glad they're making sure we are moving forward with solid releases and not just letting releases 'escape' that have significant issues.

Flex are currently saying that they're on track for an end-of-September release of the V1 software.  That's good news, but as far as I'm concerned I'd rather they continue the prerelease incremental developments until V1 is completely ready.  By that I mean fully delivering on the expected level of receiver quality for SSB, CW and AM and generally solid in terms of the client UX.  Anyway, we'll see what happens soon enough I suppose.