ShinyBroker 2: Fetch, Process, and Plot Historical Market Data

ShinyBroker is an ongoing development project at the Duke University FinTech Masters Program and is available for public use on paper accounts. Feature requests and bug reports are welcome on our GitHub page as we continue build up a full-featured platform for use in and outside the classroom.

### ShinyBroker lets you build the trading system you want in Shiny and connect it with Interactive Brokers.

This article will demonstrate how to

1. Create a new Python script in PyCharm with File > New, then select "Python File" and give it an informative name (the example uses "hello_world_1.py").
2. Paste the code below into your script:

       import shinybroker as sb

       # Create an instance of a ShinyBroker App object using the default ui and server
       app = sb.sb_app(
           host='127.0.0.1',  # localhost TWS is being served on your local machine
           port=7497,         # make this match the port in your API Settings config
           client_id=10742    # picked at random, choose another Client ID if preferred
       )

       # Run the app
       app.run()

3. Run your Python script. There are many ways to do so, but one quick and easy way is to highlight all of the code in your script with "select all", right-click the selection, and choose "Execute Selection in Python Console". It's even faster with keyboard shortcuts, e.g., for a Mac: command-A to select all, then alt-shift-E to execute. You can find those commands, and their shortcuts, by right-clicking on highlighted code in PyCharm.
4. After you execute the code PyCharm will launch a new Python console to run the code you told it to run. Your screen should look something like the below. You can see what the code you ran did: it created a ShinyBroker app object and told it where to find an IBKR connection by specifying `host`, `port`, and `client_id`. By itself, that `app` object does nothing but sit there in variable space; to make it run, you have to call its `run` method, which we did in the final line of the sample code when we called `app.run()`. That command is the one that started the continuously-running Python process that you see in your Console. That process serves your ShinyBroker app at the blue hyperlink appearing in the command line output. The app will continue to run until you tell it to stop (e.g., by clicking the red square in the Python Console in PyCharm).
5. To view your app, click the hyperlink in the command line output. Doing so will open your first ShinyBroker App in your default browser! You can also copy-paste the url into a browser of your choice. (And if you're thinking can I serve the app on one computer and access the app via browser from another computer/phone in a different location the answer is "yes, you can", but that's a later topic)

Interact with the app and make note of a few important points.

When you navigate a browser to the url at which your app is being served, you should be greeted with an app that looks something like the screen snip below.

Take a few minutes to poke around in the app and try out some of the default tabs & tools provided for tasks involving market data, contract matching, socket messages from IBKR, and so on.

Especially important to note is the section on the home page, which currently reads `no ui passed to sb_ui()`. That space is special -- it is reserved for you, the trader, to put UI elements and controls for your trading system. Since no ui was specified when we created this simple app object in this hello world example, ShinyBroker is telling you that it's empty.

We'll conclude this tutorial in the next step by injecting a little bit of ui into that space.

When you're finished exploring the app and ready to move on, stop the app from running by clicking the red square button in the console tab in PyCharm.

You're now ready for the final step in this tutorial in which you'll add a bit of live, functional UI to your ShinyBroker app.

If you're new to Shiny and reactive programming it is very highly recommended that you brush up on the basics by reviewing some of the very accessible and informative videos that the team at Posit has created to teach this topic. The video tutorial below, by Winston Chang of Posit, is an excellent place to start. As you watch, remember that anything you can build in Shiny, you can use in ShinyBroker.

Now it's time for the real learning & takeaway part of this turtorial. Let's take a minute to understand and appreciate that all user interfaces must necessarily operate on the same basic principle:

1. Some kind of new information becomes available as input; for example: user keystrokes, the market price of an asset changes, an order fills, etc. No matter what its form or purpose, the point is that some kind of change has taken place meaning that there is new, or, equivalently, updated data available that is of importance to you, the trader. Information like this shows up asynchronously, meaning that there is no way to tell when or if it will become available. Therefore a system like ShinyBroker must always be on the ball and ready to handle incoming data as it comes in from the brokerage connection.
2. Incoming data is collected by a server function, whose job it is to process the data into a useful return value and pass that value to the appropriate place in your app. For example, the input data might be a time-indexed list containing the past hour's prices for a stock at 1-minute intervals, and a server function's job might be to accept that list as input and return the timestamp of the highest (or lowest, etc.) price during the period.
3. The server function may pass its return value to just about anything the trader can dream up but its ultimate fate is to be displayed to the trader as some sort of output that is understandable to the trader; for example: a plot, an updated forecast calculation, or a text box.

This tutorial culminates in the next step, which will show you how to implement this process using the magic of Shiny, which was written to greatly simplify the entire input -> processing -> output loop, making these kinds of systems much easier for the trader to build.

Our newly added ui will cause the ShinyBroker app to operate as follows:

1. The user inputs a string to an input text box
2. A server function watches the contents of the text box and when it changes, it appends the string "You entered '" to the beginning of the input's text, and appends "'." to the end; therefore, the resulting string is "You entered '{string that you entered}'.".
3. The return string is passed to an output text box where you, the trader, can read it.

To accomplish this feat we'll need to add 3 new elements to our ShinyBroker app:

1. A text input box so that the user has something to type into
2. A server function that performs the string appending steps
3. A text output that displays the final string

You can build just about anything you want starting from this humble example simply by keeping the "input -> processing -> output" data flow in your mind and making intellegent use of the buttons, widgets, plots, and other Python tools that are readily available in Shiny and other Python packages.

Please try the next step now.

First, we'll need the two pieces of UI that will appear on the web page: an input for the user to type into, and an output to see the result. We'll use the `input_text` and `output_code` objects made available by the `ui` object in Shiny, and store them together in a `div`, with the input on top of the output. All sorts of different components are available both in Shiny and in other packages like shinywidgets, or you can build your own. In this example, however, we'll only need two components: `input_text` and `output_code`. In the code below we use these components to create a new piece of ui and assign to it the appropriate name `a_piece_of_new_ui`.

    from shiny import ui

    # Some UI to add to the app
    a_piece_of_new_ui = ui.div(
        ui.input_text(
            id='sb_example_text_in',
            label='Example Input. Type something!'
        ),
        ui.output_code('sb_example_text_out')
    )

Now we need some logic: a function that recognizes the input text box (which we named `sb_example_text_in` above) as an input that it needs to watch. It should also possess the additional properties such that, whenever the text in the input changes, the function executes, calculates the new string that we want for the text output, and sends that value to the output object that we named `sb_example_text_out`. ShinyBroker uses the Shiny framework (see the video above) to simplify all of the 'watching and updating' tasks in order to make it easy to write a function that does the job.

In Shiny-speak, what we want to write is called a rendering function, a special type of function that was created for exactly the kind of 'watching and updating' functionality that we need. A suitable rendering function is provided below but if you're new to Shiny, consider the following bullet points to help you understand rendering functions and why the code is written the way it is so that you can be better equipped to write your own.

- Shiny takes into consideration the names you give to your server functions and uses them to figure out where the return values should be passed. In most use cases, including the current tutorial, that's a lot simpler than it sounds because it boils down to: when you write your function, just give it the same name as the output you want it to update. That simple 'rule' will get you through 95% of your use cases. In our case, we want to update the output object named `sb_example_text_out`, so that's the name we'll assign to our function.
- We want the function to run every time the value of the input box `sb_example_text_in` changes, so somewhere within the function's body we'll need to call `input.sb_example_text_in()` to fetch the contents of the input box. Shiny handles the rest, which is extremely convenient because it allows you to treat `input .sb_example_text_in()` almost like a variable that you can always trust to contain "whatever is currently in the input box". Note that calling an input function to get its value carries with it a computational expense, so if you find yourself making calls to the same input more than once within the same render function, it's probably bad code. You're most likely better off if you just make the call once, store the input's value as a local variable, and then use that variable in the rest of your function code.
- Finally, we want to communicate to Shiny that our function is indeed a rendering function, which we can do by decorating it with a `@render` decorator. All sorts of render decorators are available, and of course you can write your own, but we'll be using `@render.code` so that the output text looks cool and monospaced-formatted like computer code. You can try stopping and re-running your app with other decorators like `@render.text` to see what happens with other sorts of decorators.

Taking all that into account, a render function that suits our purposes might look like the one below:

        @render.code
        def sb_example_text_out():
            return f"You entered '{input.sb_example_text_in()}'."

All that's left is to put that function into an app! Fortunately, ShinyBroker makes that easy. In practice, when making a real trading app, you will probably have tons of different functions required to make the app work, perhaps stored in different files, doing tasks like calculating your models, updating your charts, outputs, and so on. No matter how many you have, you can include all your functions in your `app` simply by wrapping them into one overall server function that you can pass to `sb.sb_app()` when you build your ShinyBroker app. It does not matter what you name your server function; the only requirement is that its signature contains the five parameters `input: Inputs, output: Outputs, session: Session, ib_socket, sb_rvs`. The part of your app that you, the the trader, will write will not always, or even usually, make use all five parameters explicitly, but they need to be there in your server function so that ShinyBroker can keep track of session information in its internals.

Below, we do exactly that: wrap the render function we just wrote into a new server function named `a_server_function`.

    from shiny import Inputs, Outputs, Session, ui, render

    # Server to support the new UI
    # Signature must always contain the following five parameters:
    #   input, output, session, ib_socket, and sb_rvs
    def a_server_function(
            input: Inputs, output: Outputs, session: Session, ib_socket, sb_rvs
    ):
        @render.code
        def sb_example_text_out():
            return f"You entered '{input.sb_example_text_in()}'."

And finally... here comes the magic of ShinyBroker. We can create a new ShinyBroker app by passing our ui and server functions as arguments to `sb_app`:

    import shinybroker as sb

    # Create a ShinyBroker app with the new ui and server
    app = sb.sb_app(
        a_piece_of_new_ui,
        a_server_function,
        host='127.0.0.1',
        port=7497,
        client_id=10742
    )

And that's it! Altogether, the fully completed code example, including `app.run()`, can be found below:

    import shinybroker as sb
    from shiny import Inputs, Outputs, Session, ui, render

    # Some UI to add to the app
    a_piece_of_new_ui = ui.div(
        ui.input_text(
            id='sb_example_text_in',
            label='Example Input. Type something!'
        ),
        ui.output_code('sb_example_text_out')
    )

    # Server to support the new UI
    # Signature must always contain the following five parameters:
    #   input, output, session, ib_socket, and sb_rvs
    def a_server_function(
            input: Inputs, output: Outputs, session: Session, ib_socket, sb_rvs
    ):
        @render.code
        def sb_example_text_out():
            return f"You entered '{input.sb_example_text_in()}'."

    # Create a ShinyBroker app with the new ui and server
    app = sb.sb_app(
        a_piece_of_new_ui,
        a_server_function,
        host='127.0.0.1',
        port=7497,
        client_id=10742
    )

    app.run()

Run your app and view it in a browser like you did in Step 4. You should see something like the below in your Home tab, and the output should update with whatever you type into the input:

This concludes the first ShinyBroker Tutorial! To recap, we:

1. Performed our installation and setup
2. Ran a test app
3. Demonstrated how the UI and Server functions that you write in your Shiny apps can be very easily integrated into IBKR's trading system with ShinyBroker

In the next installment, we'll be fetching, analyzing, and charting market data.